Jul 29 02:33:24.582303 ip-10-0-141-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 02:33:24.582321 ip-10-0-141-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 02:33:24.582331 ip-10-0-141-222 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 02:33:24.582654 ip-10-0-141-222 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 02:33:34.705981 ip-10-0-141-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 02:33:34.706000 ip-10-0-141-222 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 66c45de23f784c41a54ff166ecd739e1 -- Jul 29 02:35:09.830282 ip-10-0-141-222 systemd[1]: Starting Kubernetes Kubelet... Jul 29 02:35:10.210979 ip-10-0-141-222 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 02:35:10.210979 ip-10-0-141-222 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 02:35:10.210979 ip-10-0-141-222 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 02:35:10.210979 ip-10-0-141-222 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 02:35:10.210979 ip-10-0-141-222 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 02:35:10.213045 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.212974 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 02:35:10.218787 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218773 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 02:35:10.218787 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218786 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218790 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218793 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218796 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218804 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218807 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218810 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218812 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218815 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218820 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218824 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218827 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218830 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218833 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218836 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218838 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218841 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218844 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218846 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218849 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 02:35:10.218853 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218851 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218854 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218857 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218859 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218862 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218864 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218868 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218872 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218875 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218879 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218882 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218885 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218888 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218890 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218893 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218896 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218898 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218902 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218905 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 02:35:10.219362 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218908 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218911 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218914 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218916 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218919 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218921 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218924 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218926 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218928 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218931 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218933 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218935 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218938 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218940 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218943 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218947 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218949 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218952 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218954 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218957 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 02:35:10.219796 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218959 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218962 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218964 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218967 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218969 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218971 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218974 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218976 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218979 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218981 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218984 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218987 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218990 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218992 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218995 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.218997 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219000 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219002 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219005 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219007 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 02:35:10.220285 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219010 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219026 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219031 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219034 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219038 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219042 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219045 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219049 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219052 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219054 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219057 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219060 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219062 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219065 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219067 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219070 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219072 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219075 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219077 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 02:35:10.220761 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219717 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219723 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219726 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219728 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219731 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219734 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219737 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219739 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219742 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219745 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219747 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219752 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219756 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219759 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219762 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219765 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219768 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219770 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219773 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 02:35:10.221211 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219776 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219778 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219781 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219783 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219786 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219788 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219790 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219793 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219795 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219798 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219801 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219803 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219806 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219808 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219811 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219813 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219815 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219818 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219820 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219823 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 02:35:10.221671 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219825 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219828 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219831 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219833 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219836 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219838 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219841 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219843 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219846 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219851 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219854 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219856 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219859 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219862 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219864 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219867 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219869 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219872 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219874 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219876 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 02:35:10.222157 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219879 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219881 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219884 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219886 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219888 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219891 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219893 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219895 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219898 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219901 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219903 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219906 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219908 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219912 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219914 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219917 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219919 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219922 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219925 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 02:35:10.222639 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219927 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219930 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219933 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219935 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219938 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219940 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219943 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219945 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219947 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219950 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219952 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219955 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219958 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219960 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219963 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219965 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219967 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219970 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219972 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219976 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 02:35:10.223116 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.219979 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220338 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220343 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220347 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220350 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220356 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220359 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220362 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220364 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220367 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220369 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220372 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220375 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220377 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220380 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220382 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220385 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220388 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220390 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220393 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 02:35:10.223589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220395 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220398 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220400 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220404 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220407 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220409 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220412 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220414 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220417 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220420 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220422 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220425 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220427 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220431 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220433 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220436 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220438 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220441 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220444 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220446 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 02:35:10.224070 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220449 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220451 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220454 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220456 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220459 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220461 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220464 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220467 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220469 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220472 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220475 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220479 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220482 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220484 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220487 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220491 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220493 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220496 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220498 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220501 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 02:35:10.224581 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220503 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220505 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220508 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220510 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220513 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220515 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220517 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220521 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220525 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220528 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220532 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220535 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220539 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220542 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220545 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220548 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220551 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220554 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220556 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 02:35:10.225060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220559 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220561 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220563 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220566 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220568 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220571 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220573 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220576 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220579 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220582 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220585 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220587 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220590 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220592 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220594 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220597 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220599 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220602 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220604 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 02:35:10.225519 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220607 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.220609 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220665 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220671 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220676 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220681 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220685 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220690 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220695 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220701 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220705 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220710 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220715 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220718 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220721 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220724 2579 flags.go:64] FLAG: --cgroup-root="" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220727 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220730 2579 flags.go:64] FLAG: --client-ca-file="" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220732 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220735 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220740 2579 flags.go:64] FLAG: --cluster-domain="" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220746 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220749 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 02:35:10.226060 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220752 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220755 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220758 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220761 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220764 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220767 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220770 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220775 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220778 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220781 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220783 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220787 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220790 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220793 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220795 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220798 2579 flags.go:64] FLAG: --enable-server="true" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220801 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220805 2579 flags.go:64] FLAG: --event-burst="100" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220808 2579 flags.go:64] FLAG: --event-qps="50" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220811 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220813 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220816 2579 flags.go:64] FLAG: --eviction-hard="" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220819 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220822 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220825 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 02:35:10.226911 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220827 2579 flags.go:64] FLAG: --eviction-soft="" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220830 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220833 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220836 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220838 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220842 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220845 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220847 2579 flags.go:64] FLAG: --feature-gates="" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220851 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220854 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220856 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220859 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220862 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220865 2579 flags.go:64] FLAG: --help="false" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220869 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220872 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220874 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220877 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220880 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220884 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220886 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220889 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220892 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 02:35:10.227550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220894 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220897 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220900 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220903 2579 flags.go:64] FLAG: --kube-reserved="" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220906 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220908 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220911 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220914 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220917 2579 flags.go:64] FLAG: --lock-file="" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220919 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220922 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220925 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220930 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220932 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220935 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220938 2579 flags.go:64] FLAG: --logging-format="text" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220941 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220944 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220946 2579 flags.go:64] FLAG: --manifest-url="" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220949 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220953 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220956 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220959 2579 flags.go:64] FLAG: --max-pods="110" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220963 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220966 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 02:35:10.228139 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220969 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220971 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220974 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220977 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220980 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220986 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220989 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220992 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220995 2579 flags.go:64] FLAG: --pod-cidr="" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.220998 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221003 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221005 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221009 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221023 2579 flags.go:64] FLAG: --port="10250" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221026 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221029 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-056753c33396f95c6" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221032 2579 flags.go:64] FLAG: --qos-reserved="" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221035 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221038 2579 flags.go:64] FLAG: --register-node="true" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221041 2579 flags.go:64] FLAG: --register-with-taints="" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221044 2579 flags.go:64] FLAG: --registry-burst="10" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221047 2579 flags.go:64] FLAG: --registry-qps="5" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221050 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221053 2579 flags.go:64] FLAG: --reserved-memory="" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221056 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 02:35:10.228725 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221059 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221062 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221065 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221068 2579 flags.go:64] FLAG: --runonce="false" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221071 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221074 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221078 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221081 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221083 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221086 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221089 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221092 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221095 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221097 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221100 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221103 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221106 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221109 2579 flags.go:64] FLAG: --system-cgroups="" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221112 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221117 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221120 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221123 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221127 2579 flags.go:64] FLAG: --tls-min-version="" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221129 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221132 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 02:35:10.229385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221135 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221137 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221140 2579 flags.go:64] FLAG: --v="2" Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221144 2579 flags.go:64] FLAG: --version="false" Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221147 2579 flags.go:64] FLAG: --vmodule="" Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221151 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221154 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221267 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221271 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221274 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221277 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221280 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221283 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221287 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221290 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221292 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221295 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221298 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221300 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221303 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221305 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 02:35:10.230028 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221308 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221310 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221313 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221315 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221318 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221321 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221323 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221326 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221328 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221331 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221333 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221335 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221338 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221340 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221344 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221347 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221350 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221352 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221355 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221357 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 02:35:10.230594 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221360 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221362 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221365 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221367 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221371 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221373 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221376 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221380 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221383 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221386 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221389 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221391 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221394 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221396 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221399 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221401 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221404 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221407 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221410 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 02:35:10.231090 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221413 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221416 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221419 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221422 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221425 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221427 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221430 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221434 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221436 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221439 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221441 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221444 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221446 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221449 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221451 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221453 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221457 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221462 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221464 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221467 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 02:35:10.231547 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221470 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221472 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221474 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221477 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221480 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221482 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221484 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221487 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221489 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221492 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221494 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221497 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221499 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221502 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221505 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221507 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221509 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221512 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221514 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221518 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 02:35:10.232025 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221521 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221523 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221525 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221528 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221530 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.221533 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.221537 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.228568 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.228581 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.229397 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.229408 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229458 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229463 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229466 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 02:35:10.232515 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229469 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229472 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229474 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229477 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229480 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229482 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229485 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229487 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229490 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229493 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229495 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229497 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229500 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229503 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229505 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229507 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229510 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229514 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229518 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229521 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 02:35:10.232849 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229524 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229526 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229529 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229532 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229535 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229538 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229541 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229543 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229546 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229549 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229551 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229554 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229556 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229560 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229563 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229566 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229568 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229571 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229573 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229576 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 02:35:10.233400 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229580 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229582 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229585 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229588 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229591 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229594 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229596 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229600 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229603 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229606 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229608 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229610 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229613 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229615 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229617 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229620 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229622 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229624 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229627 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229629 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 02:35:10.233877 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229632 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229634 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229637 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229639 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229642 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229644 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229646 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229649 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229651 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229654 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229656 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229659 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229661 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229664 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229666 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229668 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229671 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229674 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229676 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 02:35:10.234357 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229679 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229681 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229684 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229686 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229688 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229691 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229693 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229696 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229698 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229700 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229703 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229705 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229707 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229710 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229712 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229722 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 02:35:10.234802 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229724 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.229729 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229835 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229840 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229843 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229846 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229848 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229850 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229853 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229856 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229858 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229861 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229863 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229865 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229868 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 02:35:10.235205 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229871 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229873 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229876 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229878 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229881 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229883 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229885 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229888 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229890 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229893 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229896 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229899 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229903 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229905 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229908 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229910 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229913 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229915 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229918 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229920 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 02:35:10.235565 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229922 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229925 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229927 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229930 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229932 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229934 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229937 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229939 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229942 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229944 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229946 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229949 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229955 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229958 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229960 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229962 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229965 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229967 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229970 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229972 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 02:35:10.236051 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229975 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229977 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229980 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229982 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229985 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229987 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229990 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229992 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229994 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229997 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.229999 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230002 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230004 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230006 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230009 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230025 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230028 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230031 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230033 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 02:35:10.236518 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230036 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230039 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230044 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230046 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230049 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230051 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230055 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230058 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230060 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230063 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230065 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230067 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230070 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230074 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230078 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230080 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230083 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230086 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230089 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 02:35:10.236954 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230092 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230095 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230099 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230101 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230104 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230107 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230109 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:10.230112 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.230116 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.230202 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.233112 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.234320 2579 server.go:1019] "Starting client certificate rotation" Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.234415 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 02:35:10.237456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.234445 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 02:35:10.237778 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.236392 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 02:35:10.239047 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.239035 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 02:35:10.250811 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.250001 2579 log.go:25] "Validated CRI v1 runtime API" Jul 29 02:35:10.252674 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.252660 2579 log.go:25] "Validated CRI v1 image API" Jul 29 02:35:10.253388 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.253374 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 02:35:10.255604 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.255588 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b616541b-b11c-4d00-bcc2-3792ae62cd60:/dev/nvme0n1p3 fccf9505-eb0a-40d0-a621-98d11161dcad:/dev/nvme0n1p4] Jul 29 02:35:10.255660 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.255604 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 02:35:10.258268 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.258247 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 02:35:10.260606 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.260513 2579 manager.go:216] Machine: {Timestamp:2026-07-29 02:35:10.259231547 +0000 UTC m=+0.319706922 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200225 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254810d44e124053599e1bceef416f SystemUUID:ec254810-d44e-1240-5359-9e1bceef416f BootID:66c45de2-3f78-4c41-a54f-f166ecd739e1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2c:d9:d7:de:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2c:d9:d7:de:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:8e:06:64:7c:1c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 02:35:10.260606 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.260602 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 02:35:10.260709 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.260683 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 02:35:10.263941 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.263922 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 02:35:10.264072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.263943 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 02:35:10.264645 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.264637 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 02:35:10.264679 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.264648 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 02:35:10.264679 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.264655 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 02:35:10.265092 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.265083 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 02:35:10.265833 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.265824 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 02:35:10.267554 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267543 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 29 02:35:10.267649 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267641 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 02:35:10.267689 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267665 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 29 02:35:10.267689 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267674 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 02:35:10.267775 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267691 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 02:35:10.267775 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267699 2579 kubelet.go:398] "Adding apiserver pod source" Jul 29 02:35:10.267775 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267707 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 02:35:10.267972 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267957 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 02:35:10.268038 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.267983 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 02:35:10.270285 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.270270 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 02:35:10.271991 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.271979 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 02:35:10.272049 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.272008 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 02:35:10.273481 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273470 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 02:35:10.273511 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273485 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 02:35:10.273511 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273492 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 02:35:10.273511 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273497 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 02:35:10.273511 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273506 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 02:35:10.273511 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273511 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 02:35:10.273637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273518 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 02:35:10.273637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273523 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 02:35:10.273637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273529 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 02:35:10.273637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273533 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 02:35:10.273637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273547 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 02:35:10.273637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.273552 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 02:35:10.274308 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.274300 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 02:35:10.274308 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.274309 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 02:35:10.277161 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.277145 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 02:35:10.277200 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.277191 2579 server.go:1268] "Started kubelet" Jul 29 02:35:10.277312 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.277290 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 02:35:10.277312 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.277299 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 02:35:10.277428 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.277320 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 02:35:10.277802 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.277764 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 02:35:10.277873 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.277826 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 02:35:10.277839 ip-10-0-141-222 systemd[1]: Started Kubernetes Kubelet. Jul 29 02:35:10.279688 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.279669 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4rqn" Jul 29 02:35:10.279854 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.279835 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 29 02:35:10.280182 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.280167 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 02:35:10.282374 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.281463 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-222.ec2.internal.18c6a214473c55af default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-222.ec2.internal,UID:ip-10-0-141-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-222.ec2.internal,},FirstTimestamp:2026-07-29 02:35:10.277162415 +0000 UTC m=+0.337637793,LastTimestamp:2026-07-29 02:35:10.277162415 +0000 UTC m=+0.337637793,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-222.ec2.internal,}" Jul 29 02:35:10.282640 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.282612 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 02:35:10.284829 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.284814 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4rqn" Jul 29 02:35:10.286501 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.286488 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 02:35:10.286501 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.286495 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 02:35:10.287223 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.287074 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 02:35:10.287300 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.287226 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 02:35:10.287300 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.287239 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.287478 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.287461 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 02:35:10.288051 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288036 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 29 02:35:10.288143 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288134 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 29 02:35:10.288229 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288150 2579 factory.go:55] Registering systemd factory Jul 29 02:35:10.288297 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288288 2579 factory.go:223] Registration of the systemd container factory successfully Jul 29 02:35:10.288653 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288637 2579 factory.go:153] Registering CRI-O factory Jul 29 02:35:10.288730 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288656 2579 factory.go:223] Registration of the crio container factory successfully Jul 29 02:35:10.288730 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288701 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 02:35:10.288820 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288725 2579 factory.go:103] Registering Raw factory Jul 29 02:35:10.288820 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.288747 2579 manager.go:1196] Started watching for new ooms in manager Jul 29 02:35:10.289165 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.289148 2579 manager.go:349] Starting recovery of all containers Jul 29 02:35:10.291042 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.291001 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 02:35:10.291132 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.291073 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 02:35:10.298803 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.298789 2579 manager.go:354] Recovery completed Jul 29 02:35:10.302848 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.302833 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 02:35:10.306142 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.306125 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory" Jul 29 02:35:10.306218 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.306158 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 02:35:10.306218 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.306172 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID" Jul 29 02:35:10.306704 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.306688 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 02:35:10.306704 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.306703 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 02:35:10.306815 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.306718 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 29 02:35:10.306815 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.306721 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 02:35:10.307466 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.307451 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory" Jul 29 02:35:10.307535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.307478 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 02:35:10.307535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.307489 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID" Jul 29 02:35:10.308400 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.308390 2579 policy_none.go:49] "None policy: Start" Jul 29 02:35:10.308436 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.308404 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 02:35:10.308436 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.308413 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 02:35:10.309333 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.309321 2579 policy_none.go:47] "Start" Jul 29 02:35:10.332660 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.332568 2579 manager.go:337] "Starting Device Plugin manager" Jul 29 02:35:10.332791 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.332677 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 02:35:10.332791 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.332689 2579 server.go:87] "Starting device plugin registration server" Jul 29 02:35:10.332902 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.332890 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 02:35:10.332944 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.332904 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 02:35:10.333390 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.333096 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 02:35:10.333390 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.333168 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 02:35:10.333390 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.333176 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 02:35:10.333616 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.333460 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 02:35:10.333616 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.333507 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.346933 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.346903 2579 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2613 into the system container \"/system.slice\": " Jul 29 02:35:10.433394 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.433375 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 02:35:10.434445 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.434097 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory" Jul 29 02:35:10.434445 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.434124 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 02:35:10.434445 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.434137 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID" Jul 29 02:35:10.434445 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.434160 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.442250 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.442238 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.442294 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.442255 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-222.ec2.internal\": node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.444995 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.444969 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 02:35:10.446108 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.446084 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 02:35:10.446197 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.446121 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 02:35:10.446197 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.446141 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 02:35:10.446197 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.446172 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 02:35:10.447987 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.447971 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 02:35:10.457571 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.457551 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.547046 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.547006 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal"] Jul 29 02:35:10.547107 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.547077 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 02:35:10.547934 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.547918 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory" Jul 29 02:35:10.548009 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.547943 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 02:35:10.548009 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.547953 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID" Jul 29 02:35:10.549378 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.549356 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 02:35:10.549580 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.549557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.549641 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.549595 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 02:35:10.550444 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.550427 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory" Jul 29 02:35:10.550524 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.550455 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 02:35:10.550524 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.550465 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID" Jul 29 02:35:10.550524 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.550432 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory" Jul 29 02:35:10.550615 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.550554 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 02:35:10.550615 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.550567 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID" Jul 29 02:35:10.551559 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.551545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.551614 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.551567 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 02:35:10.552266 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.552244 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientMemory" Jul 29 02:35:10.552327 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.552268 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 02:35:10.552327 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.552277 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeHasSufficientPID" Jul 29 02:35:10.558432 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.558417 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.566543 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.566525 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-222.ec2.internal\" not found" node="ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.570649 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.570631 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-222.ec2.internal\" not found" node="ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.589491 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.589472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186a5659ac1354fa97c957b0c0a77cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"186a5659ac1354fa97c957b0c0a77cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.589571 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.589498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186a5659ac1354fa97c957b0c0a77cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"186a5659ac1354fa97c957b0c0a77cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.589571 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.589515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c30768f5a5da057efe9a1a98ac1fcde-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"4c30768f5a5da057efe9a1a98ac1fcde\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.658912 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.658893 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.690246 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.690225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186a5659ac1354fa97c957b0c0a77cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"186a5659ac1354fa97c957b0c0a77cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.690330 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.690254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186a5659ac1354fa97c957b0c0a77cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"186a5659ac1354fa97c957b0c0a77cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.690330 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.690290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c30768f5a5da057efe9a1a98ac1fcde-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"4c30768f5a5da057efe9a1a98ac1fcde\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.690330 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.690317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/186a5659ac1354fa97c957b0c0a77cc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"186a5659ac1354fa97c957b0c0a77cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.690442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.690330 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4c30768f5a5da057efe9a1a98ac1fcde-config\") pod \"kube-apiserver-proxy-ip-10-0-141-222.ec2.internal\" (UID: \"4c30768f5a5da057efe9a1a98ac1fcde\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.690442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.690316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/186a5659ac1354fa97c957b0c0a77cc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal\" (UID: \"186a5659ac1354fa97c957b0c0a77cc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.759522 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.759479 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.860086 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.860062 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:10.869223 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.869205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.873811 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:10.873791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" Jul 29 02:35:10.960865 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:10.960847 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.061332 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:11.061288 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.137028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.136997 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 02:35:11.161828 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:11.161808 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.233432 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.233407 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 02:35:11.233727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.233498 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 02:35:11.233727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.233546 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 02:35:11.262710 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:11.262692 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.291128 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.291104 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 02:30:10 +0000 UTC" deadline="2028-05-05 00:01:13.455135022 +0000 UTC" Jul 29 02:35:11.291128 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.291126 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15501h26m2.164011484s" Jul 29 02:35:11.363410 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:11.363366 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.416895 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:11.416867 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c30768f5a5da057efe9a1a98ac1fcde.slice/crio-b46293bbd0d1615aa6ad178eec5223747333632354700c6db3927090d4791840 WatchSource:0}: Error finding container b46293bbd0d1615aa6ad178eec5223747333632354700c6db3927090d4791840: Status 404 returned error can't find the container with id b46293bbd0d1615aa6ad178eec5223747333632354700c6db3927090d4791840 Jul 29 02:35:11.417315 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:11.417296 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186a5659ac1354fa97c957b0c0a77cc7.slice/crio-3cf8a0ed5c3ae18c73d2993e6104531bd1cbd0fcab93ac7d3a2fc3bf7e3abec1 WatchSource:0}: Error finding container 3cf8a0ed5c3ae18c73d2993e6104531bd1cbd0fcab93ac7d3a2fc3bf7e3abec1: Status 404 returned error can't find the container with id 3cf8a0ed5c3ae18c73d2993e6104531bd1cbd0fcab93ac7d3a2fc3bf7e3abec1 Jul 29 02:35:11.421504 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.421491 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 02:35:11.449585 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.449547 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"186a5659ac1354fa97c957b0c0a77cc7","Type":"ContainerStarted","Data":"3cf8a0ed5c3ae18c73d2993e6104531bd1cbd0fcab93ac7d3a2fc3bf7e3abec1"} Jul 29 02:35:11.450526 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.450509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" event={"ID":"4c30768f5a5da057efe9a1a98ac1fcde","Type":"ContainerStarted","Data":"b46293bbd0d1615aa6ad178eec5223747333632354700c6db3927090d4791840"} Jul 29 02:35:11.463693 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:11.463668 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.566831 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:11.566808 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.672040 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:11.671984 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-222.ec2.internal\" not found" Jul 29 02:35:11.687897 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.687877 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 02:35:11.736440 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.736416 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 02:35:11.788203 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.788176 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" Jul 29 02:35:11.798997 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.798977 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 02:35:11.800054 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.800041 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" Jul 29 02:35:11.807610 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:11.807593 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 02:35:12.269812 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.269786 2579 apiserver.go:52] "Watching apiserver" Jul 29 02:35:12.276369 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.276343 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 02:35:12.276740 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.276717 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-2rsk8","kube-system/konnectivity-agent-zmlzv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal","openshift-multus/multus-d6khs","openshift-multus/network-metrics-daemon-bn4t5","openshift-network-diagnostics/network-check-target-jz5m2","openshift-network-operator/iptables-alerter-tkkz6","kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bz26c","openshift-dns/node-resolver-h6kpk","openshift-image-registry/node-ca-gm6b5","openshift-multus/multus-additional-cni-plugins-jr8vc","openshift-ovn-kubernetes/ovnkube-node-kwqp4"] Jul 29 02:35:12.280638 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.280618 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.282378 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.282355 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 02:35:12.282573 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.282556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qs6dx\"" Jul 29 02:35:12.282656 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.282605 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 02:35:12.282973 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.282956 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.284677 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.284625 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ggbls\"" Jul 29 02:35:12.284792 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.284757 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 02:35:12.284860 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.284787 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 02:35:12.284860 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.284632 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 02:35:12.285150 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.285061 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 02:35:12.285243 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.285165 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 02:35:12.288979 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.287442 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 02:35:12.288979 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.287595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.289762 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.289333 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c5v7l\"" Jul 29 02:35:12.289762 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.289497 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 02:35:12.289762 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.289511 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 02:35:12.289762 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.289640 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 02:35:12.290754 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.290736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.292275 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.292252 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 02:35:12.292621 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.292601 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cc2vv\"" Jul 29 02:35:12.292874 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.292710 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 02:35:12.292874 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.292754 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 02:35:12.293026 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.292971 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 02:35:12.293083 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.293053 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 02:35:12.293191 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.293170 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 02:35:12.293297 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.293281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.295056 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.295020 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 02:35:12.295167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.295144 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hnpvg\"" Jul 29 02:35:12.295747 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.295732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:12.295829 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.295802 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:12.297402 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-ovnkube-script-lib\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297512 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-system-cni-dir\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.297512 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-kubelet\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297512 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-run-netns\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysctl-d\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-host\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-systemd\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-ovn\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-cni-bin\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f16b74f-467e-4774-af25-54144c4b3353-ovn-node-metrics-cert\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdlmw\" (UniqueName: \"kubernetes.io/projected/6f16b74f-467e-4774-af25-54144c4b3353-kube-api-access-sdlmw\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.297793 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-run\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-var-lib-kubelet\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-device-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-sys-fs\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-node-log\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-log-socket\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.297997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-tuned\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-tmp\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-cnibin\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-socket-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-etc-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-modprobe-d\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-sys\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxsf\" (UniqueName: \"kubernetes.io/projected/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-kube-api-access-bkxsf\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.298313 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf58k\" (UniqueName: \"kubernetes.io/projected/74141b92-5bc1-4a45-b38e-816aae3bf805-kube-api-access-lf58k\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-cni-netd\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298329 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-ovnkube-config\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-env-overrides\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-lib-modules\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-registration-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-etc-selinux\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-slash\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-kubernetes\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysctl-conf\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-os-release\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysconfig\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-systemd\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.299072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.299602 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmhx\" (UniqueName: \"kubernetes.io/projected/21e67ffb-48ba-4975-99b6-3319df78688d-kube-api-access-ktmhx\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.299602 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-systemd-units\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299602 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-var-lib-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299602 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.298892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.299946 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.299926 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkh8j\"" Jul 29 02:35:12.300127 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.300070 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 02:35:12.300237 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.300132 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 02:35:12.300237 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.300220 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 02:35:12.301216 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.301154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:12.301346 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.301223 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:12.301606 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.301589 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 02:35:12.303664 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.303647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.305152 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.305136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nqz7j\"" Jul 29 02:35:12.305371 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.305345 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 02:35:12.305450 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.305393 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 02:35:12.306531 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.306436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.306531 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.306503 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:12.309245 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.309203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.310838 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.310819 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 02:35:12.310943 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.310855 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-68mbk\"" Jul 29 02:35:12.311041 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.311027 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 02:35:12.311563 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.311546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.313297 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.313276 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 02:35:12.313378 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.313334 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 02:35:12.313505 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.313490 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 02:35:12.313840 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.313810 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dl8lt\"" Jul 29 02:35:12.326777 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.326759 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xdwdh" Jul 29 02:35:12.337170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.337147 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xdwdh" Jul 29 02:35:12.388938 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.388908 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 02:35:12.399529 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-systemd\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.399636 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96r9j\" (UniqueName: \"kubernetes.io/projected/2bbaf700-6f58-4abc-91ef-94e2e438103b-kube-api-access-96r9j\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.399636 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxklh\" (UniqueName: \"kubernetes.io/projected/8e084583-075e-40cb-93f4-afffca0b7f53-kube-api-access-rxklh\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.399636 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.399636 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-system-cni-dir\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-systemd\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-k8s-cni-cncf-io\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-cni-multus\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-system-cni-dir\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-kubelet\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysctl-d\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-kubelet\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-cni-bin\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f16b74f-467e-4774-af25-54144c4b3353-ovn-node-metrics-cert\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.399853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-run\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-tmp\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysctl-d\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-cni-bin\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-tmp-dir\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-sys-fs\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.399974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-cnibin\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc64185a-12f2-4b53-9f35-a67980be08bb-dbus\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-sys-fs\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-run\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-socket-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-cnibin\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-modprobe-d\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc64185a-12f2-4b53-9f35-a67980be08bb-kubelet-config\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-lib-modules\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-socket-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-hostroot\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.400535 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400192 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-modprobe-d\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-slash\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysctl-conf\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-slash\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-lib-modules\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-netns\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e084583-075e-40cb-93f4-afffca0b7f53-host-slash\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysconfig\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysctl-conf\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmhx\" (UniqueName: \"kubernetes.io/projected/21e67ffb-48ba-4975-99b6-3319df78688d-kube-api-access-ktmhx\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400457 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-sysconfig\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/027117dd-7fa0-4382-ac51-1ebf12758b86-serviceca\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.401365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bbaf700-6f58-4abc-91ef-94e2e438103b-cni-binary-copy\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-multus-certs\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e084583-075e-40cb-93f4-afffca0b7f53-iptables-alerter-script\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-systemd-units\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-var-lib-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-ovnkube-script-lib\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-systemd-units\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400677 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-var-lib-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/027117dd-7fa0-4382-ac51-1ebf12758b86-host\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-cni-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-os-release\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-kubelet\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128f1d0e-50e5-4717-8531-dd5aa754433d-agent-certs\") pod \"konnectivity-agent-zmlzv\" (UID: \"128f1d0e-50e5-4717-8531-dd5aa754433d\") " pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-run-netns\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-host\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-run-netns\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.402224 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbz4\" (UniqueName: \"kubernetes.io/projected/027117dd-7fa0-4382-ac51-1ebf12758b86-kube-api-access-tmbz4\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-host\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400976 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.400998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjz2\" (UniqueName: \"kubernetes.io/projected/bfbbffef-def3-42d4-b59d-0dee8edf10b0-kube-api-access-2zjz2\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-systemd\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-ovn\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdlmw\" (UniqueName: \"kubernetes.io/projected/6f16b74f-467e-4774-af25-54144c4b3353-kube-api-access-sdlmw\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-var-lib-kubelet\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-tuned\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-ovnkube-script-lib\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401503 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-systemd\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-var-lib-kubelet\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-system-cni-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.403015 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-run-ovn\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-daemon-config\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-device-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-node-log\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-log-socket\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-conf-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-node-log\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401842 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e67ffb-48ba-4975-99b6-3319df78688d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-log-socket\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-etc-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-device-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-sys\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-etc-openvswitch\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxsf\" (UniqueName: \"kubernetes.io/projected/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-kube-api-access-bkxsf\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.401994 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-sys\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.403569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kph6p\" (UniqueName: \"kubernetes.io/projected/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-kube-api-access-kph6p\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf58k\" (UniqueName: \"kubernetes.io/projected/74141b92-5bc1-4a45-b38e-816aae3bf805-kube-api-access-lf58k\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-cni-netd\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-ovnkube-config\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-env-overrides\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128f1d0e-50e5-4717-8531-dd5aa754433d-konnectivity-ca\") pod \"konnectivity-agent-zmlzv\" (UID: \"128f1d0e-50e5-4717-8531-dd5aa754433d\") " pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f16b74f-467e-4774-af25-54144c4b3353-host-cni-netd\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-hosts-file\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-registration-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-etc-selinux\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-kubernetes\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-os-release\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-cnibin\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402523 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-kubernetes\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-socket-dir-parent\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-registration-dir\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.404113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-cni-bin\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.404661 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-etc-kubernetes\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.404661 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/74141b92-5bc1-4a45-b38e-816aae3bf805-etc-selinux\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.404661 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-ovnkube-config\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.404661 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21e67ffb-48ba-4975-99b6-3319df78688d-os-release\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.404661 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.402777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f16b74f-467e-4774-af25-54144c4b3353-env-overrides\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.404661 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.404029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f16b74f-467e-4774-af25-54144c4b3353-ovn-node-metrics-cert\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.404661 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.404283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-tmp\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.404865 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.404771 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-etc-tuned\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.411318 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.411276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmhx\" (UniqueName: \"kubernetes.io/projected/21e67ffb-48ba-4975-99b6-3319df78688d-kube-api-access-ktmhx\") pod \"multus-additional-cni-plugins-jr8vc\" (UID: \"21e67ffb-48ba-4975-99b6-3319df78688d\") " pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.412412 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.412387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxsf\" (UniqueName: \"kubernetes.io/projected/f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5-kube-api-access-bkxsf\") pod \"tuned-bz26c\" (UID: \"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5\") " pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.412511 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.412460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdlmw\" (UniqueName: \"kubernetes.io/projected/6f16b74f-467e-4774-af25-54144c4b3353-kube-api-access-sdlmw\") pod \"ovnkube-node-kwqp4\" (UID: \"6f16b74f-467e-4774-af25-54144c4b3353\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.412818 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.412797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf58k\" (UniqueName: \"kubernetes.io/projected/74141b92-5bc1-4a45-b38e-816aae3bf805-kube-api-access-lf58k\") pod \"aws-ebs-csi-driver-node-n7khn\" (UID: \"74141b92-5bc1-4a45-b38e-816aae3bf805\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.454627 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.454607 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 02:35:12.503375 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:12.503458 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph6p\" (UniqueName: \"kubernetes.io/projected/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-kube-api-access-kph6p\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.503458 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128f1d0e-50e5-4717-8531-dd5aa754433d-konnectivity-ca\") pod \"konnectivity-agent-zmlzv\" (UID: \"128f1d0e-50e5-4717-8531-dd5aa754433d\") " pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.503458 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-hosts-file\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.503580 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-cnibin\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503580 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-socket-dir-parent\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503580 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-cni-bin\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503580 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-etc-kubernetes\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503580 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96r9j\" (UniqueName: \"kubernetes.io/projected/2bbaf700-6f58-4abc-91ef-94e2e438103b-kube-api-access-96r9j\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxklh\" (UniqueName: \"kubernetes.io/projected/8e084583-075e-40cb-93f4-afffca0b7f53-kube-api-access-rxklh\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-cnibin\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-k8s-cni-cncf-io\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-cni-multus\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-tmp-dir\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc64185a-12f2-4b53-9f35-a67980be08bb-dbus\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-hosts-file\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.503748 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc64185a-12f2-4b53-9f35-a67980be08bb-kubelet-config\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-hostroot\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-netns\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e084583-075e-40cb-93f4-afffca0b7f53-host-slash\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc64185a-12f2-4b53-9f35-a67980be08bb-dbus\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/027117dd-7fa0-4382-ac51-1ebf12758b86-serviceca\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bbaf700-6f58-4abc-91ef-94e2e438103b-cni-binary-copy\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-socket-dir-parent\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-multus-certs\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e084583-075e-40cb-93f4-afffca0b7f53-iptables-alerter-script\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-cni-bin\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/027117dd-7fa0-4382-ac51-1ebf12758b86-host\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-etc-kubernetes\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.503973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-cni-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504014 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-os-release\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128f1d0e-50e5-4717-8531-dd5aa754433d-konnectivity-ca\") pod \"konnectivity-agent-zmlzv\" (UID: \"128f1d0e-50e5-4717-8531-dd5aa754433d\") " pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-kubelet\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc64185a-12f2-4b53-9f35-a67980be08bb-kubelet-config\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-cni-multus\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128f1d0e-50e5-4717-8531-dd5aa754433d-agent-certs\") pod \"konnectivity-agent-zmlzv\" (UID: \"128f1d0e-50e5-4717-8531-dd5aa754433d\") " pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-hostroot\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-netns\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbz4\" (UniqueName: \"kubernetes.io/projected/027117dd-7fa0-4382-ac51-1ebf12758b86-kube-api-access-tmbz4\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjz2\" (UniqueName: \"kubernetes.io/projected/bfbbffef-def3-42d4-b59d-0dee8edf10b0-kube-api-access-2zjz2\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-tmp-dir\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-system-cni-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-daemon-config\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/027117dd-7fa0-4382-ac51-1ebf12758b86-host\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-k8s-cni-cncf-io\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-os-release\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.504700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e084583-075e-40cb-93f4-afffca0b7f53-host-slash\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/027117dd-7fa0-4382-ac51-1ebf12758b86-serviceca\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504639 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-var-lib-kubelet\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.504679 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-host-run-multus-certs\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.504718 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504699 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-cni-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.504754 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs podName:bfbbffef-def3-42d4-b59d-0dee8edf10b0 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:13.004732916 +0000 UTC m=+3.065208300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs") pod "network-metrics-daemon-bn4t5" (UID: "bfbbffef-def3-42d4-b59d-0dee8edf10b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-system-cni-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.504829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret podName:dc64185a-12f2-4b53-9f35-a67980be08bb nodeName:}" failed. No retries permitted until 2026-07-29 02:35:13.004817396 +0000 UTC m=+3.065292762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret") pod "global-pull-secret-syncer-2rsk8" (UID: "dc64185a-12f2-4b53-9f35-a67980be08bb") : object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-conf-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.504931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-conf-dir\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.505164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bbaf700-6f58-4abc-91ef-94e2e438103b-cni-binary-copy\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.505513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.505269 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2bbaf700-6f58-4abc-91ef-94e2e438103b-multus-daemon-config\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.506008 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.505608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e084583-075e-40cb-93f4-afffca0b7f53-iptables-alerter-script\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.507063 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.507041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128f1d0e-50e5-4717-8531-dd5aa754433d-agent-certs\") pod \"konnectivity-agent-zmlzv\" (UID: \"128f1d0e-50e5-4717-8531-dd5aa754433d\") " pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.509318 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.509131 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 02:35:12.509318 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.509156 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 02:35:12.509318 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.509169 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lppfl for pod openshift-network-diagnostics/network-check-target-jz5m2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:12.509318 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:12.509260 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl podName:e6d98d91-c933-4866-a95e-4b9110aa50b1 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:13.00924993 +0000 UTC m=+3.069725294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lppfl" (UniqueName: "kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl") pod "network-check-target-jz5m2" (UID: "e6d98d91-c933-4866-a95e-4b9110aa50b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:12.512198 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.512157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph6p\" (UniqueName: \"kubernetes.io/projected/8d06fbb6-3aad-4a03-88dc-4a17c06ccaad-kube-api-access-kph6p\") pod \"node-resolver-h6kpk\" (UID: \"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad\") " pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.513386 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.513360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjz2\" (UniqueName: \"kubernetes.io/projected/bfbbffef-def3-42d4-b59d-0dee8edf10b0-kube-api-access-2zjz2\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:12.513531 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.513362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96r9j\" (UniqueName: \"kubernetes.io/projected/2bbaf700-6f58-4abc-91ef-94e2e438103b-kube-api-access-96r9j\") pod \"multus-d6khs\" (UID: \"2bbaf700-6f58-4abc-91ef-94e2e438103b\") " pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.514151 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.514131 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxklh\" (UniqueName: \"kubernetes.io/projected/8e084583-075e-40cb-93f4-afffca0b7f53-kube-api-access-rxklh\") pod \"iptables-alerter-tkkz6\" (UID: \"8e084583-075e-40cb-93f4-afffca0b7f53\") " pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.515006 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.514979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbz4\" (UniqueName: \"kubernetes.io/projected/027117dd-7fa0-4382-ac51-1ebf12758b86-kube-api-access-tmbz4\") pod \"node-ca-gm6b5\" (UID: \"027117dd-7fa0-4382-ac51-1ebf12758b86\") " pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:12.593586 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.593539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bz26c" Jul 29 02:35:12.601320 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.601285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" Jul 29 02:35:12.611447 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.611427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" Jul 29 02:35:12.617755 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.617738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:12.624570 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.624548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d6khs" Jul 29 02:35:12.632420 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.632401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tkkz6" Jul 29 02:35:12.639531 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.639501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:12.648397 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.648370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6kpk" Jul 29 02:35:12.656810 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:12.656786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gm6b5" Jul 29 02:35:13.008821 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:13.008742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:13.008821 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:13.008797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:13.009021 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.008904 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:13.009021 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.008937 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:13.009021 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.008975 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs podName:bfbbffef-def3-42d4-b59d-0dee8edf10b0 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:14.008955108 +0000 UTC m=+4.069430473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs") pod "network-metrics-daemon-bn4t5" (UID: "bfbbffef-def3-42d4-b59d-0dee8edf10b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:13.009021 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.008995 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret podName:dc64185a-12f2-4b53-9f35-a67980be08bb nodeName:}" failed. No retries permitted until 2026-07-29 02:35:14.008984868 +0000 UTC m=+4.069460238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret") pod "global-pull-secret-syncer-2rsk8" (UID: "dc64185a-12f2-4b53-9f35-a67980be08bb") : object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:13.109350 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:13.109315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:13.109506 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.109446 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 02:35:13.109506 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.109463 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 02:35:13.109506 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.109472 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lppfl for pod openshift-network-diagnostics/network-check-target-jz5m2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:13.109604 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.109517 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl podName:e6d98d91-c933-4866-a95e-4b9110aa50b1 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:14.109504135 +0000 UTC m=+4.169979500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lppfl" (UniqueName: "kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl") pod "network-check-target-jz5m2" (UID: "e6d98d91-c933-4866-a95e-4b9110aa50b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:13.122833 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:13.122807 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 02:35:13.340727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:13.340618 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 02:30:12 +0000 UTC" deadline="2028-01-28 12:02:02.051756887 +0000 UTC" Jul 29 02:35:13.340727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:13.340669 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13161h26m48.711092334s" Jul 29 02:35:13.446941 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:13.446916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:13.447071 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:13.447015 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:13.493461 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.493433 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128f1d0e_50e5_4717_8531_dd5aa754433d.slice/crio-2cf3fbcf912b8b2311e417655e11d4680a16c187576a7efa1e705840edaf8dc3 WatchSource:0}: Error finding container 2cf3fbcf912b8b2311e417655e11d4680a16c187576a7efa1e705840edaf8dc3: Status 404 returned error can't find the container with id 2cf3fbcf912b8b2311e417655e11d4680a16c187576a7efa1e705840edaf8dc3 Jul 29 02:35:13.502674 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.502652 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cb7b2b_49b4_4340_a0e1_63ab3132dbc5.slice/crio-65038cb52e0847b1d838a8d8e8c0e5f5df4fcadc15291b8bbbefdeb0910c03ce WatchSource:0}: Error finding container 65038cb52e0847b1d838a8d8e8c0e5f5df4fcadc15291b8bbbefdeb0910c03ce: Status 404 returned error can't find the container with id 65038cb52e0847b1d838a8d8e8c0e5f5df4fcadc15291b8bbbefdeb0910c03ce Jul 29 02:35:13.503903 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.503877 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74141b92_5bc1_4a45_b38e_816aae3bf805.slice/crio-a35e6f506783f5bbc0ea8e46a06b333d0cb115e9c92fb730eae2478299433dd3 WatchSource:0}: Error finding container a35e6f506783f5bbc0ea8e46a06b333d0cb115e9c92fb730eae2478299433dd3: Status 404 returned error can't find the container with id a35e6f506783f5bbc0ea8e46a06b333d0cb115e9c92fb730eae2478299433dd3 Jul 29 02:35:13.505305 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.505266 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e084583_075e_40cb_93f4_afffca0b7f53.slice/crio-5061dd7ee36a24a7c8661ad12f719196cf1616721678adefd7317ff8628a3d4f WatchSource:0}: Error finding container 5061dd7ee36a24a7c8661ad12f719196cf1616721678adefd7317ff8628a3d4f: Status 404 returned error can't find the container with id 5061dd7ee36a24a7c8661ad12f719196cf1616721678adefd7317ff8628a3d4f Jul 29 02:35:13.506297 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.506270 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbaf700_6f58_4abc_91ef_94e2e438103b.slice/crio-385e19703d6f85f776043f6f97cc55db78715fab518b86a39b2a939b742972fb WatchSource:0}: Error finding container 385e19703d6f85f776043f6f97cc55db78715fab518b86a39b2a939b742972fb: Status 404 returned error can't find the container with id 385e19703d6f85f776043f6f97cc55db78715fab518b86a39b2a939b742972fb Jul 29 02:35:13.507571 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.507493 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d06fbb6_3aad_4a03_88dc_4a17c06ccaad.slice/crio-0338edf027ff118f1285642ed2dcfa0b3a2756b194395a3627fd427ff11c87b1 WatchSource:0}: Error finding container 0338edf027ff118f1285642ed2dcfa0b3a2756b194395a3627fd427ff11c87b1: Status 404 returned error can't find the container with id 0338edf027ff118f1285642ed2dcfa0b3a2756b194395a3627fd427ff11c87b1 Jul 29 02:35:13.512047 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.509889 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e67ffb_48ba_4975_99b6_3319df78688d.slice/crio-3e43f12188b257b861a9f85d279eb911deee8d251f4e54c56ea0d770b39ca768 WatchSource:0}: Error finding container 3e43f12188b257b861a9f85d279eb911deee8d251f4e54c56ea0d770b39ca768: Status 404 returned error can't find the container with id 3e43f12188b257b861a9f85d279eb911deee8d251f4e54c56ea0d770b39ca768 Jul 29 02:35:13.512047 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:13.510793 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027117dd_7fa0_4382_ac51_1ebf12758b86.slice/crio-892ab0a9920d72114bb0e4b4f3c0c189c4bd691b32e4faa43417074f60d4ff2e WatchSource:0}: Error finding container 892ab0a9920d72114bb0e4b4f3c0c189c4bd691b32e4faa43417074f60d4ff2e: Status 404 returned error can't find the container with id 892ab0a9920d72114bb0e4b4f3c0c189c4bd691b32e4faa43417074f60d4ff2e Jul 29 02:35:14.017440 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.016572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:14.017440 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.016799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:14.017440 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.016747 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:14.017440 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.017017 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs podName:bfbbffef-def3-42d4-b59d-0dee8edf10b0 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:16.016998284 +0000 UTC m=+6.077473663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs") pod "network-metrics-daemon-bn4t5" (UID: "bfbbffef-def3-42d4-b59d-0dee8edf10b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:14.017440 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.016959 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:14.017440 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.017373 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret podName:dc64185a-12f2-4b53-9f35-a67980be08bb nodeName:}" failed. No retries permitted until 2026-07-29 02:35:16.01735706 +0000 UTC m=+6.077832423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret") pod "global-pull-secret-syncer-2rsk8" (UID: "dc64185a-12f2-4b53-9f35-a67980be08bb") : object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:14.117341 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.117301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:14.117649 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.117581 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 02:35:14.117736 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.117658 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 02:35:14.117736 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.117683 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lppfl for pod openshift-network-diagnostics/network-check-target-jz5m2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:14.117853 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.117739 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl podName:e6d98d91-c933-4866-a95e-4b9110aa50b1 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:16.117719043 +0000 UTC m=+6.178194415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lppfl" (UniqueName: "kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl") pod "network-check-target-jz5m2" (UID: "e6d98d91-c933-4866-a95e-4b9110aa50b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:14.341628 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.341541 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 02:30:12 +0000 UTC" deadline="2028-01-05 02:59:52.509393266 +0000 UTC" Jul 29 02:35:14.341628 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.341585 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12600h24m38.167812769s" Jul 29 02:35:14.453720 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.452966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:14.453720 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.453113 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:14.453720 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.453493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:14.453720 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:14.453595 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:14.461806 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.461757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bz26c" event={"ID":"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5","Type":"ContainerStarted","Data":"65038cb52e0847b1d838a8d8e8c0e5f5df4fcadc15291b8bbbefdeb0910c03ce"} Jul 29 02:35:14.467255 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.467195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmlzv" event={"ID":"128f1d0e-50e5-4717-8531-dd5aa754433d","Type":"ContainerStarted","Data":"2cf3fbcf912b8b2311e417655e11d4680a16c187576a7efa1e705840edaf8dc3"} Jul 29 02:35:14.479645 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.478908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" event={"ID":"4c30768f5a5da057efe9a1a98ac1fcde","Type":"ContainerStarted","Data":"569c0fe7ac85f189eefaa6f16cb0cd07be965cc3d39d0531597dbdc6dacf0e13"} Jul 29 02:35:14.486105 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.485654 2579 generic.go:356] "Generic (PLEG): container finished" podID="186a5659ac1354fa97c957b0c0a77cc7" containerID="cd42f2823443d47f847ef9697996f1dd899850fd302b9b5a7a672bfe32217ff5" exitCode=0 Jul 29 02:35:14.486105 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.485734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"186a5659ac1354fa97c957b0c0a77cc7","Type":"ContainerDied","Data":"cd42f2823443d47f847ef9697996f1dd899850fd302b9b5a7a672bfe32217ff5"} Jul 29 02:35:14.492798 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.492712 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"a8addb9e5878eed8ed8760e89a7d3a633f23a4392e8feb8b1fe0b5c5d09b5909"} Jul 29 02:35:14.498282 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.498104 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gm6b5" event={"ID":"027117dd-7fa0-4382-ac51-1ebf12758b86","Type":"ContainerStarted","Data":"892ab0a9920d72114bb0e4b4f3c0c189c4bd691b32e4faa43417074f60d4ff2e"} Jul 29 02:35:14.500822 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.500759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerStarted","Data":"3e43f12188b257b861a9f85d279eb911deee8d251f4e54c56ea0d770b39ca768"} Jul 29 02:35:14.506738 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.506421 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-222.ec2.internal" podStartSLOduration=3.5064052820000002 podStartE2EDuration="3.506405282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:14.492673292 +0000 UTC m=+4.553148681" watchObservedRunningTime="2026-07-29 02:35:14.506405282 +0000 UTC m=+4.566880673" Jul 29 02:35:14.509173 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.509111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkkz6" event={"ID":"8e084583-075e-40cb-93f4-afffca0b7f53","Type":"ContainerStarted","Data":"5061dd7ee36a24a7c8661ad12f719196cf1616721678adefd7317ff8628a3d4f"} Jul 29 02:35:14.525371 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.525294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" event={"ID":"74141b92-5bc1-4a45-b38e-816aae3bf805","Type":"ContainerStarted","Data":"a35e6f506783f5bbc0ea8e46a06b333d0cb115e9c92fb730eae2478299433dd3"} Jul 29 02:35:14.558412 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.558380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6kpk" event={"ID":"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad","Type":"ContainerStarted","Data":"0338edf027ff118f1285642ed2dcfa0b3a2756b194395a3627fd427ff11c87b1"} Jul 29 02:35:14.571397 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:14.571373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d6khs" event={"ID":"2bbaf700-6f58-4abc-91ef-94e2e438103b","Type":"ContainerStarted","Data":"385e19703d6f85f776043f6f97cc55db78715fab518b86a39b2a939b742972fb"} Jul 29 02:35:15.447358 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:15.446841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:15.447358 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:15.446956 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:15.585861 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:15.585483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" event={"ID":"186a5659ac1354fa97c957b0c0a77cc7","Type":"ContainerStarted","Data":"3d8f9cd7ca858798d8912ae80805d430e649149ab0893632dd99281a7ea503e9"} Jul 29 02:35:16.028974 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:16.028255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:16.028974 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:16.028297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:16.028974 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.028414 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:16.028974 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.028479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret podName:dc64185a-12f2-4b53-9f35-a67980be08bb nodeName:}" failed. No retries permitted until 2026-07-29 02:35:20.028460843 +0000 UTC m=+10.088936212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret") pod "global-pull-secret-syncer-2rsk8" (UID: "dc64185a-12f2-4b53-9f35-a67980be08bb") : object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:16.028974 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.028850 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:16.028974 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.028896 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs podName:bfbbffef-def3-42d4-b59d-0dee8edf10b0 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:20.028881108 +0000 UTC m=+10.089356474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs") pod "network-metrics-daemon-bn4t5" (UID: "bfbbffef-def3-42d4-b59d-0dee8edf10b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:16.129797 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:16.129194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:16.129797 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.129357 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 02:35:16.129797 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.129378 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 02:35:16.129797 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.129389 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lppfl for pod openshift-network-diagnostics/network-check-target-jz5m2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:16.129797 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.129448 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl podName:e6d98d91-c933-4866-a95e-4b9110aa50b1 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:20.129430583 +0000 UTC m=+10.189905974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lppfl" (UniqueName: "kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl") pod "network-check-target-jz5m2" (UID: "e6d98d91-c933-4866-a95e-4b9110aa50b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:16.452323 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:16.452257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:16.452751 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.452374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:16.452751 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:16.452728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:16.452866 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:16.452834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:17.447184 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:17.446717 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:17.447184 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:17.446839 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:18.447167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:18.447127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:18.447610 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:18.447261 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:18.447683 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:18.447637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:18.447776 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:18.447746 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:19.447069 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:19.446976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:19.447225 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:19.447110 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:20.060254 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:20.060004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:20.060254 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:20.060067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:20.060254 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.060138 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:20.060254 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.060191 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:20.060254 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.060207 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs podName:bfbbffef-def3-42d4-b59d-0dee8edf10b0 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:28.060186801 +0000 UTC m=+18.120662165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs") pod "network-metrics-daemon-bn4t5" (UID: "bfbbffef-def3-42d4-b59d-0dee8edf10b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:20.060254 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.060228 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret podName:dc64185a-12f2-4b53-9f35-a67980be08bb nodeName:}" failed. No retries permitted until 2026-07-29 02:35:28.060216735 +0000 UTC m=+18.120692098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret") pod "global-pull-secret-syncer-2rsk8" (UID: "dc64185a-12f2-4b53-9f35-a67980be08bb") : object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:20.160767 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:20.160728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:20.160907 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.160877 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 02:35:20.160907 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.160897 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 02:35:20.160968 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.160928 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lppfl for pod openshift-network-diagnostics/network-check-target-jz5m2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:20.160997 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.160987 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl podName:e6d98d91-c933-4866-a95e-4b9110aa50b1 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:28.160968476 +0000 UTC m=+18.221443845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lppfl" (UniqueName: "kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl") pod "network-check-target-jz5m2" (UID: "e6d98d91-c933-4866-a95e-4b9110aa50b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:20.450604 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:20.450510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:20.451057 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.450618 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:20.451057 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:20.450691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:20.451057 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:20.450786 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:21.446738 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:21.446705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:21.446877 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:21.446840 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:22.449880 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:22.449854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:22.449880 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:22.449863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:22.450409 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:22.449977 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:22.450409 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:22.450118 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:23.446934 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:23.446904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:23.447127 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:23.447007 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:24.446656 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:24.446627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:24.447107 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:24.446668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:24.447107 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:24.446756 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:24.447107 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:24.446899 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:25.446422 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:25.446389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:25.446584 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:25.446495 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:26.449288 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:26.449260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:26.449288 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:26.449272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:26.449752 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:26.449366 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:26.449752 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:26.449521 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:27.447207 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:27.447179 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:27.447375 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:27.447281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:28.121522 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:28.121483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:28.121955 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:28.121533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:28.121955 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.121642 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:28.121955 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.121646 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:28.121955 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.121703 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret podName:dc64185a-12f2-4b53-9f35-a67980be08bb nodeName:}" failed. No retries permitted until 2026-07-29 02:35:44.121686358 +0000 UTC m=+34.182161729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret") pod "global-pull-secret-syncer-2rsk8" (UID: "dc64185a-12f2-4b53-9f35-a67980be08bb") : object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:28.121955 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.121722 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs podName:bfbbffef-def3-42d4-b59d-0dee8edf10b0 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:44.121714057 +0000 UTC m=+34.182189425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs") pod "network-metrics-daemon-bn4t5" (UID: "bfbbffef-def3-42d4-b59d-0dee8edf10b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:28.222714 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:28.222683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:28.222858 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.222807 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 02:35:28.222858 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.222822 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 02:35:28.222858 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.222831 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lppfl for pod openshift-network-diagnostics/network-check-target-jz5m2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:28.222983 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.222876 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl podName:e6d98d91-c933-4866-a95e-4b9110aa50b1 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:44.222859739 +0000 UTC m=+34.283335102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lppfl" (UniqueName: "kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl") pod "network-check-target-jz5m2" (UID: "e6d98d91-c933-4866-a95e-4b9110aa50b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:28.447083 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:28.447000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:28.447219 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:28.447004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:28.447219 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.447127 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:28.447332 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:28.447222 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:29.447245 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:29.447213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:29.447661 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:29.447328 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:30.447978 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:30.447753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:30.447978 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:30.447859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:30.447978 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:30.447904 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:30.447978 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:30.447959 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:31.446866 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.446787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:31.446966 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:31.446896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:31.617289 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.617094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gm6b5" event={"ID":"027117dd-7fa0-4382-ac51-1ebf12758b86","Type":"ContainerStarted","Data":"a6368279115bf6917c59673b9a0a8da14e7adc6340dc0221eb996a7c4f45fc97"} Jul 29 02:35:31.620800 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.620741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" event={"ID":"74141b92-5bc1-4a45-b38e-816aae3bf805","Type":"ContainerStarted","Data":"b5aa9847d9b947a49d911307da7c18569ca1269b600ea7113fa2e91287454d53"} Jul 29 02:35:31.622494 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.622477 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6kpk" event={"ID":"8d06fbb6-3aad-4a03-88dc-4a17c06ccaad","Type":"ContainerStarted","Data":"0e0763c64ab3e26a08a0491dbe2c615e7a398628c25fd90380e24ded6a163472"} Jul 29 02:35:31.623676 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.623649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d6khs" event={"ID":"2bbaf700-6f58-4abc-91ef-94e2e438103b","Type":"ContainerStarted","Data":"7aea89ca1203a4432b9e71183566442a2027cfd15f4fbc51c423bd8fe3eeb067"} Jul 29 02:35:31.625492 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.625410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bz26c" event={"ID":"f3cb7b2b-49b4-4340-a0e1-63ab3132dbc5","Type":"ContainerStarted","Data":"3983c0689b425dfad7ff444441a906fcf6de90281a688f935bd1c4c21f29ce7d"} Jul 29 02:35:31.626794 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.626760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmlzv" event={"ID":"128f1d0e-50e5-4717-8531-dd5aa754433d","Type":"ContainerStarted","Data":"94761e9c3621679678a72cd7f5e2f58c1fc5f0797009a9c5446add43246e7b69"} Jul 29 02:35:31.630170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.630134 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-222.ec2.internal" podStartSLOduration=20.630119361 podStartE2EDuration="20.630119361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:15.599077641 +0000 UTC m=+5.659553048" watchObservedRunningTime="2026-07-29 02:35:31.630119361 +0000 UTC m=+21.690594746" Jul 29 02:35:31.630460 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.630432 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gm6b5" podStartSLOduration=3.730749713 podStartE2EDuration="21.630425712s" totalImagesPullingTime="17.899675999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:31.6299794 +0000 UTC m=+21.690454784" watchObservedRunningTime="2026-07-29 02:35:31.630425712 +0000 UTC m=+21.690901098" Jul 29 02:35:31.643291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.643237 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zmlzv" podStartSLOduration=3.7273738610000002 podStartE2EDuration="21.643225367s" totalImagesPullingTime="17.915851506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:31.642941907 +0000 UTC m=+21.703417291" watchObservedRunningTime="2026-07-29 02:35:31.643225367 +0000 UTC m=+21.703700749" Jul 29 02:35:31.654502 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.654475 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h6kpk" podStartSLOduration=7.940090908 podStartE2EDuration="21.654465675s" totalImagesPullingTime="13.714374767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:31.654352529 +0000 UTC m=+21.714827914" watchObservedRunningTime="2026-07-29 02:35:31.654465675 +0000 UTC m=+21.714941058" Jul 29 02:35:31.667591 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:31.667563 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bz26c" podStartSLOduration=3.756537433 podStartE2EDuration="21.667553066s" totalImagesPullingTime="17.911015633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:31.667529851 +0000 UTC m=+21.728005236" watchObservedRunningTime="2026-07-29 02:35:31.667553066 +0000 UTC m=+21.728028494" Jul 29 02:35:32.446700 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.446567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:32.446792 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.446628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:32.446881 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:32.446774 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:32.446921 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:32.446876 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:32.629261 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.629214 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tkkz6" event={"ID":"8e084583-075e-40cb-93f4-afffca0b7f53","Type":"ContainerStarted","Data":"00f58ca66281e10570b4638a9ac7cf4e6962fa897f780ba0027b1213259bdb52"} Jul 29 02:35:32.631810 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.631794 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:35:32.632129 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.632104 2579 generic.go:356] "Generic (PLEG): container finished" podID="6f16b74f-467e-4774-af25-54144c4b3353" containerID="8010dc26aec1db6edde9b6bc1d26d675d9c3adae56c68a171c482b377cddc839" exitCode=1 Jul 29 02:35:32.632212 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.632168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"0cc8cbfc8fc436480c051a161ed23d0dc531926c7af8623d1cf583d0e5126611"} Jul 29 02:35:32.632268 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.632218 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"157a3c54de67cd21390d9b97de37ba14d9cc52c4f58fe0e038cbee0d0f0f470f"} Jul 29 02:35:32.632268 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.632247 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"cf7b61e7eb2f154d5abd883455345f60cdc050747e355331a570346536c45691"} Jul 29 02:35:32.632338 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.632287 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"6df549f497939342056398b1bbc24ee757c1a4958397037a1b19806e2c58223b"} Jul 29 02:35:32.632338 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.632319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerDied","Data":"8010dc26aec1db6edde9b6bc1d26d675d9c3adae56c68a171c482b377cddc839"} Jul 29 02:35:32.632404 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.632365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"07524ec94447d57adb7454ec4410c79a19bc668cc244180808301e75e487998c"} Jul 29 02:35:32.633561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.633543 2579 generic.go:356] "Generic (PLEG): container finished" podID="21e67ffb-48ba-4975-99b6-3319df78688d" containerID="3295de6a788088a2ddb787834927fc16a4d7d9326e9f59790ea732416366d2f8" exitCode=0 Jul 29 02:35:32.633638 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.633603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerDied","Data":"3295de6a788088a2ddb787834927fc16a4d7d9326e9f59790ea732416366d2f8"} Jul 29 02:35:32.641678 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.641637 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tkkz6" podStartSLOduration=8.924315919 podStartE2EDuration="22.641626216s" totalImagesPullingTime="13.717310297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:32.641292022 +0000 UTC m=+22.701767409" watchObservedRunningTime="2026-07-29 02:35:32.641626216 +0000 UTC m=+22.702101599" Jul 29 02:35:32.680137 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:32.680097 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d6khs" podStartSLOduration=4.757835931 podStartE2EDuration="22.680081161s" totalImagesPullingTime="17.92224523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:32.67927074 +0000 UTC m=+22.739746124" watchObservedRunningTime="2026-07-29 02:35:32.680081161 +0000 UTC m=+22.740556549" Jul 29 02:35:33.378170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:33.377743 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 02:35:33.447068 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:33.447047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:33.447188 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:33.447163 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:33.637317 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:33.637251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" event={"ID":"74141b92-5bc1-4a45-b38e-816aae3bf805","Type":"ContainerStarted","Data":"3c3404b7e7c2bd1dacb4798df0310da9ab5d12672284c31756debf56cc150afb"} Jul 29 02:35:34.347058 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.346933 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T02:35:33.377758747Z","UUID":"32707c31-dcaa-487d-b234-370eba60676f","Handler":null,"Name":"","Endpoint":""} Jul 29 02:35:34.348502 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.348479 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 02:35:34.348613 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.348535 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 02:35:34.446699 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.446675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:34.446699 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.446685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:34.446852 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:34.446804 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:34.446977 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:34.446954 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:34.643637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.643391 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:35:34.644069 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.644047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"b9d000aa2c4984c777839d6f0dc4c8ed8de3636b208c0d02c37429495846fec2"} Jul 29 02:35:34.646227 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.646201 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" event={"ID":"74141b92-5bc1-4a45-b38e-816aae3bf805","Type":"ContainerStarted","Data":"1d6c7169efb650f0e3a316c40ee6e2244714322800800339ec0af8fabea07fb6"} Jul 29 02:35:34.660743 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:34.660697 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7khn" podStartSLOduration=4.03375998 podStartE2EDuration="24.660679699s" totalImagesPullingTime="20.626919719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:34.66020382 +0000 UTC m=+24.720679205" watchObservedRunningTime="2026-07-29 02:35:34.660679699 +0000 UTC m=+24.721155085" Jul 29 02:35:35.446374 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:35.446349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:35.446629 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:35.446449 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:35.939057 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:35.938970 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:35.939574 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:35.939539 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:36.446764 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:36.446736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:36.446764 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:36.446747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:36.446965 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:36.446855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:36.447040 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:36.446968 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:36.651430 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:36.651402 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:36.652055 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:36.652036 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmlzv" Jul 29 02:35:37.446657 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.446417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:37.447234 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:37.446770 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:37.654895 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.654810 2579 generic.go:356] "Generic (PLEG): container finished" podID="21e67ffb-48ba-4975-99b6-3319df78688d" containerID="29e5468380bd2ee4891fa36aec9e1497c7d1cde50a7591c16eef122a2454c516" exitCode=0 Jul 29 02:35:37.655077 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.654902 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerDied","Data":"29e5468380bd2ee4891fa36aec9e1497c7d1cde50a7591c16eef122a2454c516"} Jul 29 02:35:37.658507 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.658490 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:35:37.658850 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.658830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"e4bb104bc8629d34ebf5a924f9a6a72e951aa67db882fe1234b48f0f367928d2"} Jul 29 02:35:37.659145 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.659101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:37.659288 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.659275 2579 scope.go:117] "RemoveContainer" containerID="8010dc26aec1db6edde9b6bc1d26d675d9c3adae56c68a171c482b377cddc839" Jul 29 02:35:37.673102 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:37.673084 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:38.446728 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.446695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:38.447199 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:38.446802 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:38.447199 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.446855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:38.447199 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:38.446941 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:38.664851 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.664828 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:35:38.665247 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.665219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" event={"ID":"6f16b74f-467e-4774-af25-54144c4b3353","Type":"ContainerStarted","Data":"f6fed8230d68127fa2c98745c9d1225c4fdb568ed4f34783f409a2b81ef25386"} Jul 29 02:35:38.665366 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.665350 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:38.665419 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.665413 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:38.679291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.679271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:35:38.687993 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:38.687957 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" podStartSLOduration=10.416135125 podStartE2EDuration="28.687946004s" totalImagesPullingTime="18.271810879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:38.687160921 +0000 UTC m=+28.747636308" watchObservedRunningTime="2026-07-29 02:35:38.687946004 +0000 UTC m=+28.748421387" Jul 29 02:35:39.118291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.118262 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jz5m2"] Jul 29 02:35:39.118395 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.118385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:39.118512 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:39.118485 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:39.121289 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.121265 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rsk8"] Jul 29 02:35:39.121424 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.121399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:39.121537 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:39.121501 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:39.121885 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.121864 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bn4t5"] Jul 29 02:35:39.121975 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.121964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:39.122099 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:39.122079 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:39.668984 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.668927 2579 generic.go:356] "Generic (PLEG): container finished" podID="21e67ffb-48ba-4975-99b6-3319df78688d" containerID="818d82860f910c49e5580235556c1ead1014f8cd652d8977896d18c84831fe22" exitCode=0 Jul 29 02:35:39.669244 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:39.669007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerDied","Data":"818d82860f910c49e5580235556c1ead1014f8cd652d8977896d18c84831fe22"} Jul 29 02:35:40.447866 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:40.447837 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:40.448040 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:40.447920 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:40.448040 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:40.448011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:40.448142 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:40.448100 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:40.448198 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:40.448158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:40.448282 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:40.448265 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:41.675609 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:41.675580 2579 generic.go:356] "Generic (PLEG): container finished" podID="21e67ffb-48ba-4975-99b6-3319df78688d" containerID="5468bb0df0e66e153428ae148bae7908a791c682ceda74228dd71ddc2526b979" exitCode=0 Jul 29 02:35:41.676093 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:41.675619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerDied","Data":"5468bb0df0e66e153428ae148bae7908a791c682ceda74228dd71ddc2526b979"} Jul 29 02:35:42.446358 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:42.446328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:42.446524 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:42.446328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:42.446524 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:42.446441 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:42.446639 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:42.446540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:42.446639 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:42.446534 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:42.446717 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:42.446633 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:44.147248 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:44.147210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:44.147248 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:44.147252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:44.147807 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.147357 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:44.147807 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.147424 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs podName:bfbbffef-def3-42d4-b59d-0dee8edf10b0 nodeName:}" failed. No retries permitted until 2026-07-29 02:36:16.147404179 +0000 UTC m=+66.207879547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs") pod "network-metrics-daemon-bn4t5" (UID: "bfbbffef-def3-42d4-b59d-0dee8edf10b0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 02:35:44.147807 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.147348 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:44.147807 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.147469 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret podName:dc64185a-12f2-4b53-9f35-a67980be08bb nodeName:}" failed. No retries permitted until 2026-07-29 02:36:16.147460631 +0000 UTC m=+66.207935995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret") pod "global-pull-secret-syncer-2rsk8" (UID: "dc64185a-12f2-4b53-9f35-a67980be08bb") : object "kube-system"/"original-pull-secret" not registered Jul 29 02:35:44.247776 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:44.247745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:44.247939 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.247920 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 02:35:44.247996 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.247947 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 02:35:44.247996 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.247958 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lppfl for pod openshift-network-diagnostics/network-check-target-jz5m2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:44.248089 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.248031 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl podName:e6d98d91-c933-4866-a95e-4b9110aa50b1 nodeName:}" failed. No retries permitted until 2026-07-29 02:36:16.248002614 +0000 UTC m=+66.308477978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lppfl" (UniqueName: "kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl") pod "network-check-target-jz5m2" (UID: "e6d98d91-c933-4866-a95e-4b9110aa50b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 02:35:44.447167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:44.447139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:44.447167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:44.447141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:44.447362 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:44.447297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:44.447362 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.447298 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2rsk8" podUID="dc64185a-12f2-4b53-9f35-a67980be08bb" Jul 29 02:35:44.447431 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.447415 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz5m2" podUID="e6d98d91-c933-4866-a95e-4b9110aa50b1" Jul 29 02:35:44.447534 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:44.447511 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bn4t5" podUID="bfbbffef-def3-42d4-b59d-0dee8edf10b0" Jul 29 02:35:45.274432 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.274358 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-222.ec2.internal" event="NodeReady" Jul 29 02:35:45.274961 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.274489 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 02:35:45.317372 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.317343 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c688544ff-lnvqd"] Jul 29 02:35:45.319732 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.319714 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gfgld"] Jul 29 02:35:45.319853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.319840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.321494 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.321474 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qlddw"] Jul 29 02:35:45.321640 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.321615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.321869 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.321820 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9hmnf\"" Jul 29 02:35:45.322272 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.322003 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 02:35:45.322272 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.322118 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 02:35:45.322272 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.322164 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 02:35:45.323380 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.323359 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 02:35:45.323482 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.323420 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.323824 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.323763 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 02:35:45.323906 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.323830 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hdc6x\"" Jul 29 02:35:45.325430 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.325401 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v2jkq\"" Jul 29 02:35:45.325430 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.325427 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 02:35:45.325550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.325449 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 02:35:45.325550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.325505 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 02:35:45.328074 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.328047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 02:35:45.333295 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.331068 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qlddw"] Jul 29 02:35:45.335925 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.335905 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gfgld"] Jul 29 02:35:45.336627 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.336606 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c688544ff-lnvqd"] Jul 29 02:35:45.457505 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqrh\" (UniqueName: \"kubernetes.io/projected/ab1e2935-4cf7-42a8-9fc7-fc684d73378c-kube-api-access-2hqrh\") pod \"ingress-canary-qlddw\" (UID: \"ab1e2935-4cf7-42a8-9fc7-fc684d73378c\") " pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.457643 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457643 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dms5k\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-kube-api-access-dms5k\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457643 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-bound-sa-token\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457765 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86640fb2-8e21-43a1-86be-31998eecabab-config-volume\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.457765 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-certificates\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457765 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.457765 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab1e2935-4cf7-42a8-9fc7-fc684d73378c-cert\") pod \"ingress-canary-qlddw\" (UID: \"ab1e2935-4cf7-42a8-9fc7-fc684d73378c\") " pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.457901 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-installation-pull-secrets\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457901 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4j28\" (UniqueName: \"kubernetes.io/projected/86640fb2-8e21-43a1-86be-31998eecabab-kube-api-access-r4j28\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.457901 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-trusted-ca\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457901 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-ca-trust-extracted\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457901 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-image-registry-private-configuration\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.457901 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.457892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86640fb2-8e21-43a1-86be-31998eecabab-tmp-dir\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.558852 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.558786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86640fb2-8e21-43a1-86be-31998eecabab-tmp-dir\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.558963 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.558860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqrh\" (UniqueName: \"kubernetes.io/projected/ab1e2935-4cf7-42a8-9fc7-fc684d73378c-kube-api-access-2hqrh\") pod \"ingress-canary-qlddw\" (UID: \"ab1e2935-4cf7-42a8-9fc7-fc684d73378c\") " pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.558963 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.558888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.558963 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.558912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dms5k\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-kube-api-access-dms5k\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.558963 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.558949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-bound-sa-token\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.559172 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:45.559047 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 02:35:45.559172 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:45.559067 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c688544ff-lnvqd: secret "image-registry-tls" not found Jul 29 02:35:45.559172 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:45.559124 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls podName:20eef5a0-f2e5-487c-8b39-57c60fe1c9ce nodeName:}" failed. No retries permitted until 2026-07-29 02:35:46.059104143 +0000 UTC m=+36.119579525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls") pod "image-registry-7c688544ff-lnvqd" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce") : secret "image-registry-tls" not found Jul 29 02:35:45.559172 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86640fb2-8e21-43a1-86be-31998eecabab-config-volume\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559177 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86640fb2-8e21-43a1-86be-31998eecabab-tmp-dir\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-certificates\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab1e2935-4cf7-42a8-9fc7-fc684d73378c-cert\") pod \"ingress-canary-qlddw\" (UID: \"ab1e2935-4cf7-42a8-9fc7-fc684d73378c\") " pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-installation-pull-secrets\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4j28\" (UniqueName: \"kubernetes.io/projected/86640fb2-8e21-43a1-86be-31998eecabab-kube-api-access-r4j28\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:45.559319 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 02:35:45.559352 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:45.559355 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls podName:86640fb2-8e21-43a1-86be-31998eecabab nodeName:}" failed. No retries permitted until 2026-07-29 02:35:46.059343562 +0000 UTC m=+36.119818925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls") pod "dns-default-gfgld" (UID: "86640fb2-8e21-43a1-86be-31998eecabab") : secret "dns-default-metrics-tls" not found Jul 29 02:35:45.559706 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-trusted-ca\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.559706 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-ca-trust-extracted\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.559706 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-image-registry-private-configuration\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.559846 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/86640fb2-8e21-43a1-86be-31998eecabab-config-volume\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.559895 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-certificates\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.560028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.559991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-ca-trust-extracted\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.560222 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.560204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-trusted-ca\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.563722 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.563702 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-image-registry-private-configuration\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.563791 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.563754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-installation-pull-secrets\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.563829 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.563788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab1e2935-4cf7-42a8-9fc7-fc684d73378c-cert\") pod \"ingress-canary-qlddw\" (UID: \"ab1e2935-4cf7-42a8-9fc7-fc684d73378c\") " pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.567463 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.567412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-bound-sa-token\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.567696 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.567663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dms5k\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-kube-api-access-dms5k\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:45.567775 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.567735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4j28\" (UniqueName: \"kubernetes.io/projected/86640fb2-8e21-43a1-86be-31998eecabab-kube-api-access-r4j28\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:45.567775 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.567761 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqrh\" (UniqueName: \"kubernetes.io/projected/ab1e2935-4cf7-42a8-9fc7-fc684d73378c-kube-api-access-2hqrh\") pod \"ingress-canary-qlddw\" (UID: \"ab1e2935-4cf7-42a8-9fc7-fc684d73378c\") " pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.648028 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.647836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qlddw" Jul 29 02:35:45.778981 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:45.778956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qlddw"] Jul 29 02:35:45.783138 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:45.783113 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1e2935_4cf7_42a8_9fc7_fc684d73378c.slice/crio-3e9c4bce6f92db728dc94d4b614c1d420c26b8a1e3b66654d5ba331bc20e0739 WatchSource:0}: Error finding container 3e9c4bce6f92db728dc94d4b614c1d420c26b8a1e3b66654d5ba331bc20e0739: Status 404 returned error can't find the container with id 3e9c4bce6f92db728dc94d4b614c1d420c26b8a1e3b66654d5ba331bc20e0739 Jul 29 02:35:46.063858 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.063833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:46.063932 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.063870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:46.063978 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:46.063965 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 02:35:46.063978 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:46.063970 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 02:35:46.064068 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:46.063985 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c688544ff-lnvqd: secret "image-registry-tls" not found Jul 29 02:35:46.064068 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:46.064011 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls podName:86640fb2-8e21-43a1-86be-31998eecabab nodeName:}" failed. No retries permitted until 2026-07-29 02:35:47.063999056 +0000 UTC m=+37.124474420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls") pod "dns-default-gfgld" (UID: "86640fb2-8e21-43a1-86be-31998eecabab") : secret "dns-default-metrics-tls" not found Jul 29 02:35:46.064068 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:46.064043 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls podName:20eef5a0-f2e5-487c-8b39-57c60fe1c9ce nodeName:}" failed. No retries permitted until 2026-07-29 02:35:47.0640349 +0000 UTC m=+37.124510268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls") pod "image-registry-7c688544ff-lnvqd" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce") : secret "image-registry-tls" not found Jul 29 02:35:46.446558 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.446531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:35:46.446973 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.446568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:35:46.446973 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.446532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:35:46.449157 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.449134 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 02:35:46.449157 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.449145 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2t7w\"" Jul 29 02:35:46.449456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.449438 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 02:35:46.449456 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.449453 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 02:35:46.449695 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.449682 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 02:35:46.449845 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.449830 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ngnfs\"" Jul 29 02:35:46.690283 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.690259 2579 generic.go:356] "Generic (PLEG): container finished" podID="21e67ffb-48ba-4975-99b6-3319df78688d" containerID="b8314331250e6f3d2dceb1c390f9bca5860dee0f7d9f4e1da0bd05ad742fd993" exitCode=0 Jul 29 02:35:46.690410 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.690349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerDied","Data":"b8314331250e6f3d2dceb1c390f9bca5860dee0f7d9f4e1da0bd05ad742fd993"} Jul 29 02:35:46.693418 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:46.693395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qlddw" event={"ID":"ab1e2935-4cf7-42a8-9fc7-fc684d73378c","Type":"ContainerStarted","Data":"3e9c4bce6f92db728dc94d4b614c1d420c26b8a1e3b66654d5ba331bc20e0739"} Jul 29 02:35:47.020645 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.020569 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh"] Jul 29 02:35:47.022389 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.022372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" Jul 29 02:35:47.024424 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.024401 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 29 02:35:47.024520 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.024448 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 29 02:35:47.024587 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.024554 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hv5pb\"" Jul 29 02:35:47.033666 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.033639 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh"] Jul 29 02:35:47.066981 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.066959 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lkqfc"] Jul 29 02:35:47.068654 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.068640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.070360 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.070334 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 02:35:47.070360 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.070353 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 02:35:47.070515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.070405 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 02:35:47.070515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.070358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 02:35:47.070515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.070470 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6stpf\"" Jul 29 02:35:47.071781 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.071764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:47.071824 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.071802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:47.071909 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.071896 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 02:35:47.071953 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.071932 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 02:35:47.071953 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.071948 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c688544ff-lnvqd: secret "image-registry-tls" not found Jul 29 02:35:47.072007 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.071978 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls podName:86640fb2-8e21-43a1-86be-31998eecabab nodeName:}" failed. No retries permitted until 2026-07-29 02:35:49.071961749 +0000 UTC m=+39.132437114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls") pod "dns-default-gfgld" (UID: "86640fb2-8e21-43a1-86be-31998eecabab") : secret "dns-default-metrics-tls" not found Jul 29 02:35:47.072077 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.072058 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls podName:20eef5a0-f2e5-487c-8b39-57c60fe1c9ce nodeName:}" failed. No retries permitted until 2026-07-29 02:35:49.072039209 +0000 UTC m=+39.132514587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls") pod "image-registry-7c688544ff-lnvqd" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce") : secret "image-registry-tls" not found Jul 29 02:35:47.080991 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.080972 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lkqfc"] Jul 29 02:35:47.172437 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.172288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t92c\" (UniqueName: \"kubernetes.io/projected/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-api-access-4t92c\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.172561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.172446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs44s\" (UniqueName: \"kubernetes.io/projected/9e6388bc-0b0b-4183-ab8b-82a174d54462-kube-api-access-rs44s\") pod \"migrator-6865c9967-wt6rh\" (UID: \"9e6388bc-0b0b-4183-ab8b-82a174d54462\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" Jul 29 02:35:47.172561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.172466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f54510f7-6d8e-4430-948e-76c2c50a4dc7-data-volume\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.172561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.172483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f54510f7-6d8e-4430-948e-76c2c50a4dc7-crio-socket\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.172561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.172500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.172561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.172517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.274034 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.273949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t92c\" (UniqueName: \"kubernetes.io/projected/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-api-access-4t92c\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.274034 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.273988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs44s\" (UniqueName: \"kubernetes.io/projected/9e6388bc-0b0b-4183-ab8b-82a174d54462-kube-api-access-rs44s\") pod \"migrator-6865c9967-wt6rh\" (UID: \"9e6388bc-0b0b-4183-ab8b-82a174d54462\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" Jul 29 02:35:47.274232 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.274096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f54510f7-6d8e-4430-948e-76c2c50a4dc7-data-volume\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.274232 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.274135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f54510f7-6d8e-4430-948e-76c2c50a4dc7-crio-socket\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.274232 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.274164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.274232 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.274191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.274404 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.274320 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 02:35:47.274404 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.274383 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:47.774361373 +0000 UTC m=+37.834836753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : secret "insights-runtime-extractor-tls" not found Jul 29 02:35:47.274495 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.274423 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 02:35:47.274495 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.274455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f54510f7-6d8e-4430-948e-76c2c50a4dc7-crio-socket\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.274495 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.274480 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:47.774463699 +0000 UTC m=+37.834939084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : configmap "kube-rbac-proxy" not found Jul 29 02:35:47.274495 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.274473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f54510f7-6d8e-4430-948e-76c2c50a4dc7-data-volume\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.284153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.284133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t92c\" (UniqueName: \"kubernetes.io/projected/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-api-access-4t92c\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.284277 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.284258 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs44s\" (UniqueName: \"kubernetes.io/projected/9e6388bc-0b0b-4183-ab8b-82a174d54462-kube-api-access-rs44s\") pod \"migrator-6865c9967-wt6rh\" (UID: \"9e6388bc-0b0b-4183-ab8b-82a174d54462\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" Jul 29 02:35:47.332712 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.332689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" Jul 29 02:35:47.699981 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.699950 2579 generic.go:356] "Generic (PLEG): container finished" podID="21e67ffb-48ba-4975-99b6-3319df78688d" containerID="bfb464832979ed0f662b134c41df6167374418cbfb8b1d2657231286e1beea08" exitCode=0 Jul 29 02:35:47.700451 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.700002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerDied","Data":"bfb464832979ed0f662b134c41df6167374418cbfb8b1d2657231286e1beea08"} Jul 29 02:35:47.778608 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.778580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.778742 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.778615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:47.778742 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.778707 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 02:35:47.778847 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.778780 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:48.778758751 +0000 UTC m=+38.839234120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : configmap "kube-rbac-proxy" not found Jul 29 02:35:47.778847 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.778714 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 02:35:47.778943 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:47.778859 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:48.778843016 +0000 UTC m=+38.839318389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : secret "insights-runtime-extractor-tls" not found Jul 29 02:35:47.976164 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:47.976142 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh"] Jul 29 02:35:47.979298 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:47.979272 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6388bc_0b0b_4183_ab8b_82a174d54462.slice/crio-320182e08ab03dd1356bdf49fdb737ddd99cd0dbf3c6cca25abf77acf92d6058 WatchSource:0}: Error finding container 320182e08ab03dd1356bdf49fdb737ddd99cd0dbf3c6cca25abf77acf92d6058: Status 404 returned error can't find the container with id 320182e08ab03dd1356bdf49fdb737ddd99cd0dbf3c6cca25abf77acf92d6058 Jul 29 02:35:48.703919 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:48.703668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qlddw" event={"ID":"ab1e2935-4cf7-42a8-9fc7-fc684d73378c","Type":"ContainerStarted","Data":"90cb7e3165c202587cdb865ef52c337f3ae66d24f4d39b37333024482ecd0295"} Jul 29 02:35:48.704900 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:48.704873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" event={"ID":"9e6388bc-0b0b-4183-ab8b-82a174d54462","Type":"ContainerStarted","Data":"320182e08ab03dd1356bdf49fdb737ddd99cd0dbf3c6cca25abf77acf92d6058"} Jul 29 02:35:48.708724 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:48.708701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" event={"ID":"21e67ffb-48ba-4975-99b6-3319df78688d","Type":"ContainerStarted","Data":"c7f6b18a7c4532606efd1bea58f45b72a0f9cf56c27e45316801c806b664c269"} Jul 29 02:35:48.718852 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:48.718808 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qlddw" podStartSLOduration=1.5591219600000001 podStartE2EDuration="3.718790289s" totalImagesPullingTime="2.159668329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:48.71792312 +0000 UTC m=+38.778398524" watchObservedRunningTime="2026-07-29 02:35:48.718790289 +0000 UTC m=+38.779265674" Jul 29 02:35:48.739864 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:48.739819 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jr8vc" podStartSLOduration=9.485603013 podStartE2EDuration="38.73979992s" totalImagesPullingTime="29.254196907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:48.739258042 +0000 UTC m=+38.799733437" watchObservedRunningTime="2026-07-29 02:35:48.73979992 +0000 UTC m=+38.800275307" Jul 29 02:35:48.786935 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:48.786911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:48.787076 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:48.786979 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 02:35:48.787076 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:48.787049 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:50.78703152 +0000 UTC m=+40.847506884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : configmap "kube-rbac-proxy" not found Jul 29 02:35:48.787076 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:48.787066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:48.787241 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:48.787194 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 02:35:48.787241 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:48.787230 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:50.787219956 +0000 UTC m=+40.847695323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : secret "insights-runtime-extractor-tls" not found Jul 29 02:35:49.089537 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:49.089509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:49.089705 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:49.089547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:49.089705 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:49.089627 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 02:35:49.089705 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:49.089667 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 02:35:49.089705 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:49.089689 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c688544ff-lnvqd: secret "image-registry-tls" not found Jul 29 02:35:49.089705 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:49.089696 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls podName:86640fb2-8e21-43a1-86be-31998eecabab nodeName:}" failed. No retries permitted until 2026-07-29 02:35:53.08968479 +0000 UTC m=+43.150160156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls") pod "dns-default-gfgld" (UID: "86640fb2-8e21-43a1-86be-31998eecabab") : secret "dns-default-metrics-tls" not found Jul 29 02:35:49.089896 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:49.089737 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls podName:20eef5a0-f2e5-487c-8b39-57c60fe1c9ce nodeName:}" failed. No retries permitted until 2026-07-29 02:35:53.089720914 +0000 UTC m=+43.150196289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls") pod "image-registry-7c688544ff-lnvqd" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce") : secret "image-registry-tls" not found Jul 29 02:35:49.476317 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:49.476290 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6kpk_8d06fbb6-3aad-4a03-88dc-4a17c06ccaad/dns-node-resolver/0.log" Jul 29 02:35:50.079010 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.078988 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4d2p5"] Jul 29 02:35:50.093469 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.093426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4d2p5"] Jul 29 02:35:50.093555 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.093510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.095225 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.095206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j7892\"" Jul 29 02:35:50.095324 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.095206 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 29 02:35:50.095850 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.095830 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 29 02:35:50.095942 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.095902 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 29 02:35:50.096128 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.096111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 29 02:35:50.196533 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.196511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rff\" (UniqueName: \"kubernetes.io/projected/d47540b7-d73a-4964-b8e2-6d7378f75b00-kube-api-access-c6rff\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.196616 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.196551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d47540b7-d73a-4964-b8e2-6d7378f75b00-signing-key\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.196616 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.196567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d47540b7-d73a-4964-b8e2-6d7378f75b00-signing-cabundle\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.276764 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.276744 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gm6b5_027117dd-7fa0-4382-ac51-1ebf12758b86/node-ca/0.log" Jul 29 02:35:50.297023 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.296995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rff\" (UniqueName: \"kubernetes.io/projected/d47540b7-d73a-4964-b8e2-6d7378f75b00-kube-api-access-c6rff\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.297073 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.297049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d47540b7-d73a-4964-b8e2-6d7378f75b00-signing-key\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.297107 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.297076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d47540b7-d73a-4964-b8e2-6d7378f75b00-signing-cabundle\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.297723 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.297703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d47540b7-d73a-4964-b8e2-6d7378f75b00-signing-cabundle\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.300173 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.300158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d47540b7-d73a-4964-b8e2-6d7378f75b00-signing-key\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.305170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.305150 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rff\" (UniqueName: \"kubernetes.io/projected/d47540b7-d73a-4964-b8e2-6d7378f75b00-kube-api-access-c6rff\") pod \"service-ca-658d7b7c78-4d2p5\" (UID: \"d47540b7-d73a-4964-b8e2-6d7378f75b00\") " pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.402168 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.402126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" Jul 29 02:35:50.583596 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.583562 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4d2p5"] Jul 29 02:35:50.586721 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:35:50.586697 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47540b7_d73a_4964_b8e2_6d7378f75b00.slice/crio-7bbe7c950d159dea7caf570a71bfcd96013339a9a2ecca62e8345ff7f259bd3f WatchSource:0}: Error finding container 7bbe7c950d159dea7caf570a71bfcd96013339a9a2ecca62e8345ff7f259bd3f: Status 404 returned error can't find the container with id 7bbe7c950d159dea7caf570a71bfcd96013339a9a2ecca62e8345ff7f259bd3f Jul 29 02:35:50.714115 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.714092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" event={"ID":"9e6388bc-0b0b-4183-ab8b-82a174d54462","Type":"ContainerStarted","Data":"27283327d91abb63db2cd285020ddbf51890299fdabb0426b94daa291c62b225"} Jul 29 02:35:50.714204 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.714142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" event={"ID":"9e6388bc-0b0b-4183-ab8b-82a174d54462","Type":"ContainerStarted","Data":"5b3651553efe3eafc551305307d9fbd0b13330a2691c7e11d9c5c7742732efd9"} Jul 29 02:35:50.715147 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.715123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" event={"ID":"d47540b7-d73a-4964-b8e2-6d7378f75b00","Type":"ContainerStarted","Data":"7bbe7c950d159dea7caf570a71bfcd96013339a9a2ecca62e8345ff7f259bd3f"} Jul 29 02:35:50.728369 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.728330 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wt6rh" podStartSLOduration=1.8773331610000001 podStartE2EDuration="3.728317294s" totalImagesPullingTime="1.850984133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:50.727057154 +0000 UTC m=+40.787532535" watchObservedRunningTime="2026-07-29 02:35:50.728317294 +0000 UTC m=+40.788792679" Jul 29 02:35:50.802409 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.802387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:50.802503 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:50.802413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:50.802503 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:50.802476 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 02:35:50.802599 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:50.802512 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:54.802500624 +0000 UTC m=+44.862975987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : configmap "kube-rbac-proxy" not found Jul 29 02:35:50.802599 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:50.802573 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 02:35:50.802748 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:50.802629 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:35:54.80261562 +0000 UTC m=+44.863090983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : secret "insights-runtime-extractor-tls" not found Jul 29 02:35:51.077672 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:51.077585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qlddw_ab1e2935-4cf7-42a8-9fc7-fc684d73378c/serve-healthcheck-canary/0.log" Jul 29 02:35:51.480510 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:51.480485 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wt6rh_9e6388bc-0b0b-4183-ab8b-82a174d54462/migrator/0.log" Jul 29 02:35:51.677817 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:51.677787 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wt6rh_9e6388bc-0b0b-4183-ab8b-82a174d54462/graceful-termination/0.log" Jul 29 02:35:53.123276 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:53.123218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:35:53.123573 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:53.123317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:35:53.123573 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:53.123330 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 02:35:53.123573 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:53.123413 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c688544ff-lnvqd: secret "image-registry-tls" not found Jul 29 02:35:53.123573 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:53.123379 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 02:35:53.123573 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:53.123460 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls podName:20eef5a0-f2e5-487c-8b39-57c60fe1c9ce nodeName:}" failed. No retries permitted until 2026-07-29 02:36:01.123444425 +0000 UTC m=+51.183919792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls") pod "image-registry-7c688544ff-lnvqd" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce") : secret "image-registry-tls" not found Jul 29 02:35:53.123573 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:53.123482 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls podName:86640fb2-8e21-43a1-86be-31998eecabab nodeName:}" failed. No retries permitted until 2026-07-29 02:36:01.123471606 +0000 UTC m=+51.183946970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls") pod "dns-default-gfgld" (UID: "86640fb2-8e21-43a1-86be-31998eecabab") : secret "dns-default-metrics-tls" not found Jul 29 02:35:53.724631 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:53.724593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" event={"ID":"d47540b7-d73a-4964-b8e2-6d7378f75b00","Type":"ContainerStarted","Data":"5ae2a470f80038f1da7038131298e48a09768a14e5545cee6f24dd050db20c5a"} Jul 29 02:35:53.739318 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:53.739239 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-4d2p5" podStartSLOduration=1.513012561 podStartE2EDuration="3.739222529s" totalImagesPullingTime="2.226209968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:35:53.739181835 +0000 UTC m=+43.799657222" watchObservedRunningTime="2026-07-29 02:35:53.739222529 +0000 UTC m=+43.799697914" Jul 29 02:35:54.834802 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:54.834760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:54.834802 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:54.834804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:35:54.835322 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:54.834899 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 02:35:54.835322 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:35:54.834952 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls podName:f54510f7-6d8e-4430-948e-76c2c50a4dc7 nodeName:}" failed. No retries permitted until 2026-07-29 02:36:02.834936287 +0000 UTC m=+52.895411650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lkqfc" (UID: "f54510f7-6d8e-4430-948e-76c2c50a4dc7") : secret "insights-runtime-extractor-tls" not found Jul 29 02:35:54.835428 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:35:54.835356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f54510f7-6d8e-4430-948e-76c2c50a4dc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:36:01.181985 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.181957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:36:01.182342 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.181996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:36:01.184206 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.184180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/86640fb2-8e21-43a1-86be-31998eecabab-metrics-tls\") pod \"dns-default-gfgld\" (UID: \"86640fb2-8e21-43a1-86be-31998eecabab\") " pod="openshift-dns/dns-default-gfgld" Jul 29 02:36:01.184317 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.184277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"image-registry-7c688544ff-lnvqd\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:36:01.236246 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.236223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:36:01.242440 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.242421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfgld" Jul 29 02:36:01.383245 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.383215 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c688544ff-lnvqd"] Jul 29 02:36:01.386331 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:01.386308 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eef5a0_f2e5_487c_8b39_57c60fe1c9ce.slice/crio-3ea1689310c4189e48c900b76116f6e38f37b5160d79e49aeecba19a0fda1c60 WatchSource:0}: Error finding container 3ea1689310c4189e48c900b76116f6e38f37b5160d79e49aeecba19a0fda1c60: Status 404 returned error can't find the container with id 3ea1689310c4189e48c900b76116f6e38f37b5160d79e49aeecba19a0fda1c60 Jul 29 02:36:01.396349 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.396324 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gfgld"] Jul 29 02:36:01.398733 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:01.398712 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86640fb2_8e21_43a1_86be_31998eecabab.slice/crio-d966b4aaf362958d20e3249c35e4647e3cb03742f868ce63166df4adc5c906d4 WatchSource:0}: Error finding container d966b4aaf362958d20e3249c35e4647e3cb03742f868ce63166df4adc5c906d4: Status 404 returned error can't find the container with id d966b4aaf362958d20e3249c35e4647e3cb03742f868ce63166df4adc5c906d4 Jul 29 02:36:01.746250 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.746113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfgld" event={"ID":"86640fb2-8e21-43a1-86be-31998eecabab","Type":"ContainerStarted","Data":"d966b4aaf362958d20e3249c35e4647e3cb03742f868ce63166df4adc5c906d4"} Jul 29 02:36:01.748483 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.748458 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" event={"ID":"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce","Type":"ContainerStarted","Data":"3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac"} Jul 29 02:36:01.748614 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.748502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" event={"ID":"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce","Type":"ContainerStarted","Data":"3ea1689310c4189e48c900b76116f6e38f37b5160d79e49aeecba19a0fda1c60"} Jul 29 02:36:01.748614 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.748591 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:36:01.770472 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:01.770423 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" podStartSLOduration=23.770405907 podStartE2EDuration="23.770405907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:01.76872971 +0000 UTC m=+51.829205094" watchObservedRunningTime="2026-07-29 02:36:01.770405907 +0000 UTC m=+51.830881293" Jul 29 02:36:02.894444 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:02.894395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:36:02.896493 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:02.896476 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f54510f7-6d8e-4430-948e-76c2c50a4dc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lkqfc\" (UID: \"f54510f7-6d8e-4430-948e-76c2c50a4dc7\") " pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:36:02.976389 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:02.976360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lkqfc" Jul 29 02:36:03.508241 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:03.508205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lkqfc"] Jul 29 02:36:03.520895 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:03.520869 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54510f7_6d8e_4430_948e_76c2c50a4dc7.slice/crio-954b27a020d0c2116439732efca3c11152f0bf9b49f32749a76ad8efd4b92f72 WatchSource:0}: Error finding container 954b27a020d0c2116439732efca3c11152f0bf9b49f32749a76ad8efd4b92f72: Status 404 returned error can't find the container with id 954b27a020d0c2116439732efca3c11152f0bf9b49f32749a76ad8efd4b92f72 Jul 29 02:36:03.756803 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:03.756770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfgld" event={"ID":"86640fb2-8e21-43a1-86be-31998eecabab","Type":"ContainerStarted","Data":"10814104bc355b367646681e5d7dc80da6dcac16f365b673ed5189597c58d894"} Jul 29 02:36:03.756948 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:03.756833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfgld" event={"ID":"86640fb2-8e21-43a1-86be-31998eecabab","Type":"ContainerStarted","Data":"18f68560e7205c67263a94c11dbc83925e2672d3e367d38dd282a33571310e14"} Jul 29 02:36:03.756948 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:03.756890 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gfgld" Jul 29 02:36:03.757853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:03.757833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkqfc" event={"ID":"f54510f7-6d8e-4430-948e-76c2c50a4dc7","Type":"ContainerStarted","Data":"954b27a020d0c2116439732efca3c11152f0bf9b49f32749a76ad8efd4b92f72"} Jul 29 02:36:03.773386 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:03.773349 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gfgld" podStartSLOduration=16.714503544 podStartE2EDuration="18.77333411s" totalImagesPullingTime="2.058830566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:03.771435214 +0000 UTC m=+53.831910597" watchObservedRunningTime="2026-07-29 02:36:03.77333411 +0000 UTC m=+53.833809492" Jul 29 02:36:08.440390 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.440361 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch"] Jul 29 02:36:08.442329 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.442314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.443776 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.443754 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v"] Jul 29 02:36:08.444260 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.444238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 02:36:08.444344 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.444249 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 02:36:08.444737 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.444718 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 02:36:08.444830 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.444818 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 02:36:08.445767 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.445752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.447581 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.447560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 02:36:08.447815 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.447795 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 02:36:08.447909 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.447816 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 02:36:08.447909 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.447802 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 02:36:08.458899 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.458883 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch"] Jul 29 02:36:08.462778 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.462761 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c688544ff-lnvqd"] Jul 29 02:36:08.476424 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.476407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v"] Jul 29 02:36:08.536883 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.536857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rpv\" (UniqueName: \"kubernetes.io/projected/76476d6a-c970-4770-a1f8-4e11cc16dc0c-kube-api-access-h5rpv\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.536977 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.536905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-ca\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.537072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.536966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.537072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.537009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/76476d6a-c970-4770-a1f8-4e11cc16dc0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.537072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.537049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1529bb0-b879-4768-b35c-cb8c6fb03a26-tmp\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.537188 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.537161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.537233 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.537193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6tw8\" (UniqueName: \"kubernetes.io/projected/a1529bb0-b879-4768-b35c-cb8c6fb03a26-kube-api-access-m6tw8\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.537277 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.537264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a1529bb0-b879-4768-b35c-cb8c6fb03a26-klusterlet-config\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.537322 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.537301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-hub\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.539655 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.539634 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76c7df8f66-j5gz5"] Jul 29 02:36:08.541790 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.541768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.557723 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.557705 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c7df8f66-j5gz5"] Jul 29 02:36:08.638290 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-hub\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.638477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-registry-tls\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.638477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/901124bb-feb2-4e60-aa16-749b7bfdb0e0-ca-trust-extracted\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.638477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rpv\" (UniqueName: \"kubernetes.io/projected/76476d6a-c970-4770-a1f8-4e11cc16dc0c-kube-api-access-h5rpv\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.638477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/901124bb-feb2-4e60-aa16-749b7bfdb0e0-registry-certificates\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.638477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-ca\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.638477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/76476d6a-c970-4770-a1f8-4e11cc16dc0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1529bb0-b879-4768-b35c-cb8c6fb03a26-tmp\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/901124bb-feb2-4e60-aa16-749b7bfdb0e0-trusted-ca\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-bound-sa-token\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6tw8\" (UniqueName: \"kubernetes.io/projected/a1529bb0-b879-4768-b35c-cb8c6fb03a26-kube-api-access-m6tw8\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/901124bb-feb2-4e60-aa16-749b7bfdb0e0-installation-pull-secrets\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/901124bb-feb2-4e60-aa16-749b7bfdb0e0-image-registry-private-configuration\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a1529bb0-b879-4768-b35c-cb8c6fb03a26-klusterlet-config\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.638807 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.638795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csszc\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-kube-api-access-csszc\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.639791 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.639398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1529bb0-b879-4768-b35c-cb8c6fb03a26-tmp\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.639791 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.639776 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/76476d6a-c970-4770-a1f8-4e11cc16dc0c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.642582 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.642558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-ca\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.643124 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.643104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a1529bb0-b879-4768-b35c-cb8c6fb03a26-klusterlet-config\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.643240 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.643156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-hub\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.643939 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.643917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.644862 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.644818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76476d6a-c970-4770-a1f8-4e11cc16dc0c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.646648 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.646625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rpv\" (UniqueName: \"kubernetes.io/projected/76476d6a-c970-4770-a1f8-4e11cc16dc0c-kube-api-access-h5rpv\") pod \"cluster-proxy-proxy-agent-664fc7bd59-6998v\" (UID: \"76476d6a-c970-4770-a1f8-4e11cc16dc0c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.646732 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.646654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6tw8\" (UniqueName: \"kubernetes.io/projected/a1529bb0-b879-4768-b35c-cb8c6fb03a26-kube-api-access-m6tw8\") pod \"klusterlet-addon-workmgr-7547644b45-gggch\" (UID: \"a1529bb0-b879-4768-b35c-cb8c6fb03a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.739804 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/901124bb-feb2-4e60-aa16-749b7bfdb0e0-installation-pull-secrets\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.739804 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/901124bb-feb2-4e60-aa16-749b7bfdb0e0-image-registry-private-configuration\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.739970 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csszc\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-kube-api-access-csszc\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.739970 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-registry-tls\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.739970 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/901124bb-feb2-4e60-aa16-749b7bfdb0e0-ca-trust-extracted\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.739970 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/901124bb-feb2-4e60-aa16-749b7bfdb0e0-registry-certificates\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.739970 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/901124bb-feb2-4e60-aa16-749b7bfdb0e0-trusted-ca\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.740230 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.739993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-bound-sa-token\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.741279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.740965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/901124bb-feb2-4e60-aa16-749b7bfdb0e0-ca-trust-extracted\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.741279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.741031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/901124bb-feb2-4e60-aa16-749b7bfdb0e0-registry-certificates\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.742262 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.742238 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/901124bb-feb2-4e60-aa16-749b7bfdb0e0-trusted-ca\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.742553 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.742498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/901124bb-feb2-4e60-aa16-749b7bfdb0e0-image-registry-private-configuration\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.743666 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.743645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/901124bb-feb2-4e60-aa16-749b7bfdb0e0-installation-pull-secrets\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.744226 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.744196 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-registry-tls\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.747153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.747132 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csszc\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-kube-api-access-csszc\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.749567 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.749546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/901124bb-feb2-4e60-aa16-749b7bfdb0e0-bound-sa-token\") pod \"image-registry-76c7df8f66-j5gz5\" (UID: \"901124bb-feb2-4e60-aa16-749b7bfdb0e0\") " pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.752581 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.752391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:08.769593 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.769450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" Jul 29 02:36:08.780605 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.780486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkqfc" event={"ID":"f54510f7-6d8e-4430-948e-76c2c50a4dc7","Type":"ContainerStarted","Data":"5463bf91f68b82cfeb5d380fb9872b8f80ed03a82c36bf233cfebb6f77d8fd61"} Jul 29 02:36:08.851960 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.851703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:08.916489 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.915863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch"] Jul 29 02:36:08.919876 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:08.919318 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1529bb0_b879_4768_b35c_cb8c6fb03a26.slice/crio-91b2c88f8100a7d210258614d96b60b176191b422e7fc189d2147fbd522431f7 WatchSource:0}: Error finding container 91b2c88f8100a7d210258614d96b60b176191b422e7fc189d2147fbd522431f7: Status 404 returned error can't find the container with id 91b2c88f8100a7d210258614d96b60b176191b422e7fc189d2147fbd522431f7 Jul 29 02:36:08.938141 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:08.938114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v"] Jul 29 02:36:08.944395 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:08.944349 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76476d6a_c970_4770_a1f8_4e11cc16dc0c.slice/crio-ee2dffeeba1743c1e5d15fbaa4a535febaa0f228a3684d19a9cb800c78bce9e6 WatchSource:0}: Error finding container ee2dffeeba1743c1e5d15fbaa4a535febaa0f228a3684d19a9cb800c78bce9e6: Status 404 returned error can't find the container with id ee2dffeeba1743c1e5d15fbaa4a535febaa0f228a3684d19a9cb800c78bce9e6 Jul 29 02:36:09.008179 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:09.008069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c7df8f66-j5gz5"] Jul 29 02:36:09.011929 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:09.011889 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901124bb_feb2_4e60_aa16_749b7bfdb0e0.slice/crio-cefd7dfc051a2bc67eaf067abbf33e8ba0e04e8e05cb36c844df78b15442c155 WatchSource:0}: Error finding container cefd7dfc051a2bc67eaf067abbf33e8ba0e04e8e05cb36c844df78b15442c155: Status 404 returned error can't find the container with id cefd7dfc051a2bc67eaf067abbf33e8ba0e04e8e05cb36c844df78b15442c155 Jul 29 02:36:09.785080 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:09.785044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" event={"ID":"901124bb-feb2-4e60-aa16-749b7bfdb0e0","Type":"ContainerStarted","Data":"338f3bdf301dcfc19453d078d6075457895785a105966c1c99a523d0701cf063"} Jul 29 02:36:09.785465 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:09.785102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" event={"ID":"901124bb-feb2-4e60-aa16-749b7bfdb0e0","Type":"ContainerStarted","Data":"cefd7dfc051a2bc67eaf067abbf33e8ba0e04e8e05cb36c844df78b15442c155"} Jul 29 02:36:09.785465 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:09.785148 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:09.786446 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:09.786418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" event={"ID":"a1529bb0-b879-4768-b35c-cb8c6fb03a26","Type":"ContainerStarted","Data":"91b2c88f8100a7d210258614d96b60b176191b422e7fc189d2147fbd522431f7"} Jul 29 02:36:09.787565 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:09.787539 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" event={"ID":"76476d6a-c970-4770-a1f8-4e11cc16dc0c","Type":"ContainerStarted","Data":"ee2dffeeba1743c1e5d15fbaa4a535febaa0f228a3684d19a9cb800c78bce9e6"} Jul 29 02:36:09.807675 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:09.807630 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" podStartSLOduration=1.807618857 podStartE2EDuration="1.807618857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:09.806960713 +0000 UTC m=+59.867436100" watchObservedRunningTime="2026-07-29 02:36:09.807618857 +0000 UTC m=+59.868094233" Jul 29 02:36:10.681912 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:10.681883 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwqp4" Jul 29 02:36:12.802813 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:12.802734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkqfc" event={"ID":"f54510f7-6d8e-4430-948e-76c2c50a4dc7","Type":"ContainerStarted","Data":"0883a2fc06430cc8656bc777271f9789336e829ea11eeef9f72719d595306fdd"} Jul 29 02:36:13.764965 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:13.764928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gfgld" Jul 29 02:36:16.205652 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.205610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:36:16.206043 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.205691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:36:16.207784 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.207759 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 02:36:16.207929 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.207766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 02:36:16.218482 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.218461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbbffef-def3-42d4-b59d-0dee8edf10b0-metrics-certs\") pod \"network-metrics-daemon-bn4t5\" (UID: \"bfbbffef-def3-42d4-b59d-0dee8edf10b0\") " pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:36:16.218590 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.218569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc64185a-12f2-4b53-9f35-a67980be08bb-original-pull-secret\") pod \"global-pull-secret-syncer-2rsk8\" (UID: \"dc64185a-12f2-4b53-9f35-a67980be08bb\") " pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:36:16.306761 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.306734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:36:16.308740 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.308719 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 02:36:16.318584 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.318566 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 02:36:16.329920 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.329898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppfl\" (UniqueName: \"kubernetes.io/projected/e6d98d91-c933-4866-a95e-4b9110aa50b1-kube-api-access-lppfl\") pod \"network-check-target-jz5m2\" (UID: \"e6d98d91-c933-4866-a95e-4b9110aa50b1\") " pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:36:16.460369 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.460315 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2t7w\"" Jul 29 02:36:16.466294 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.466274 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ngnfs\"" Jul 29 02:36:16.468683 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.468667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bn4t5" Jul 29 02:36:16.469863 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.469846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2rsk8" Jul 29 02:36:16.475940 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:16.475914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:36:18.206063 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.205911 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jz5m2"] Jul 29 02:36:18.210490 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:18.210459 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d98d91_c933_4866_a95e_4b9110aa50b1.slice/crio-9c19f4c1165f33abefe21667c8b98794085e8ebcdd50e2b7d06d701496776ffc WatchSource:0}: Error finding container 9c19f4c1165f33abefe21667c8b98794085e8ebcdd50e2b7d06d701496776ffc: Status 404 returned error can't find the container with id 9c19f4c1165f33abefe21667c8b98794085e8ebcdd50e2b7d06d701496776ffc Jul 29 02:36:18.218694 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.218647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bn4t5"] Jul 29 02:36:18.223026 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:18.222985 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbbffef_def3_42d4_b59d_0dee8edf10b0.slice/crio-2495d6e36dd949f7803a56943b13cf590f68f6714e69a723f56efee1b4f0e0d9 WatchSource:0}: Error finding container 2495d6e36dd949f7803a56943b13cf590f68f6714e69a723f56efee1b4f0e0d9: Status 404 returned error can't find the container with id 2495d6e36dd949f7803a56943b13cf590f68f6714e69a723f56efee1b4f0e0d9 Jul 29 02:36:18.241000 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.240975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2rsk8"] Jul 29 02:36:18.244589 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:18.244524 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc64185a_12f2_4b53_9f35_a67980be08bb.slice/crio-ff16306323098031f5d4b125424a38273d9b040955c44936502d456174c55ce1 WatchSource:0}: Error finding container ff16306323098031f5d4b125424a38273d9b040955c44936502d456174c55ce1: Status 404 returned error can't find the container with id ff16306323098031f5d4b125424a38273d9b040955c44936502d456174c55ce1 Jul 29 02:36:18.468104 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.468078 2579 patch_prober.go:28] interesting pod/image-registry-7c688544ff-lnvqd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 02:36:18.468221 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.468117 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" podUID="20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 02:36:18.835252 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.835213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lkqfc" event={"ID":"f54510f7-6d8e-4430-948e-76c2c50a4dc7","Type":"ContainerStarted","Data":"fc653f79a3b73fb22792e8fa56f5103a3088c4628d078fd3f4ed34ce09137aa3"} Jul 29 02:36:18.838410 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.838379 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bn4t5" event={"ID":"bfbbffef-def3-42d4-b59d-0dee8edf10b0","Type":"ContainerStarted","Data":"2495d6e36dd949f7803a56943b13cf590f68f6714e69a723f56efee1b4f0e0d9"} Jul 29 02:36:18.841627 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.841594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" event={"ID":"a1529bb0-b879-4768-b35c-cb8c6fb03a26","Type":"ContainerStarted","Data":"5ef8a14f615cdfceefd7ef0175b7fdaa2b271dac36a45536821d4302ffdad00e"} Jul 29 02:36:18.841826 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.841801 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:18.844170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.844131 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" Jul 29 02:36:18.845583 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.845322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" event={"ID":"76476d6a-c970-4770-a1f8-4e11cc16dc0c","Type":"ContainerStarted","Data":"5b9bf9e4bac3d7fb21932ad818924d5b5c99156dc06587f337fddf225a907470"} Jul 29 02:36:18.845583 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.845356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" event={"ID":"76476d6a-c970-4770-a1f8-4e11cc16dc0c","Type":"ContainerStarted","Data":"efdc95aaa3e11228b55e642cdde3b79feb23dbdf2e1f004655805d2df7a5a523"} Jul 29 02:36:18.845583 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.845375 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" event={"ID":"76476d6a-c970-4770-a1f8-4e11cc16dc0c","Type":"ContainerStarted","Data":"c555dbe89241d62596b65da942a610fe17b3864afab4e3c36b13dc1b799ba16c"} Jul 29 02:36:18.847221 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.847196 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jz5m2" event={"ID":"e6d98d91-c933-4866-a95e-4b9110aa50b1","Type":"ContainerStarted","Data":"9c19f4c1165f33abefe21667c8b98794085e8ebcdd50e2b7d06d701496776ffc"} Jul 29 02:36:18.849153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.849130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rsk8" event={"ID":"dc64185a-12f2-4b53-9f35-a67980be08bb","Type":"ContainerStarted","Data":"ff16306323098031f5d4b125424a38273d9b040955c44936502d456174c55ce1"} Jul 29 02:36:18.855226 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.854947 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lkqfc" podStartSLOduration=17.448575915 podStartE2EDuration="31.854929838s" totalImagesPullingTime="14.406353923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:18.851624332 +0000 UTC m=+68.912099719" watchObservedRunningTime="2026-07-29 02:36:18.854929838 +0000 UTC m=+68.915405224" Jul 29 02:36:18.868294 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.867831 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-664fc7bd59-6998v" podStartSLOduration=1.756952181 podStartE2EDuration="10.867816429s" totalImagesPullingTime="9.110864248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:18.866495222 +0000 UTC m=+68.926970608" watchObservedRunningTime="2026-07-29 02:36:18.867816429 +0000 UTC m=+68.928291816" Jul 29 02:36:18.881055 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:18.880710 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7547644b45-gggch" podStartSLOduration=1.742121081 podStartE2EDuration="10.880696809s" totalImagesPullingTime="9.138575728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:18.879682355 +0000 UTC m=+68.940157740" watchObservedRunningTime="2026-07-29 02:36:18.880696809 +0000 UTC m=+68.941172195" Jul 29 02:36:19.855374 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:19.855333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bn4t5" event={"ID":"bfbbffef-def3-42d4-b59d-0dee8edf10b0","Type":"ContainerStarted","Data":"316d51e8eaac0d1b0ca4a0e8ec3b5c3817a1c1e78092c86516674e9908e185ed"} Jul 29 02:36:19.855844 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:19.855408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bn4t5" event={"ID":"bfbbffef-def3-42d4-b59d-0dee8edf10b0","Type":"ContainerStarted","Data":"4618a334ac8930f0e58aa3815881c3495f6928486dfd55c883b041b51bce0dc3"} Jul 29 02:36:19.870140 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:19.870089 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bn4t5" podStartSLOduration=68.881793106 podStartE2EDuration="1m9.870069661s" totalImagesPullingTime="988.276555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:19.869410975 +0000 UTC m=+69.929886360" watchObservedRunningTime="2026-07-29 02:36:19.870069661 +0000 UTC m=+69.930545049" Jul 29 02:36:22.867031 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:22.866935 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jz5m2" event={"ID":"e6d98d91-c933-4866-a95e-4b9110aa50b1","Type":"ContainerStarted","Data":"4c8c6913a78a622a12df370e88677e2258a031a9e1270506683899a6fe54a6c2"} Jul 29 02:36:22.867431 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:22.867066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:36:22.868384 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:22.868362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2rsk8" event={"ID":"dc64185a-12f2-4b53-9f35-a67980be08bb","Type":"ContainerStarted","Data":"73874d74b54015f3819c32b38cb332e3756ace43d046bd1a22551261c5754f8d"} Jul 29 02:36:22.883852 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:22.883810 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jz5m2" podStartSLOduration=68.499916381 podStartE2EDuration="1m12.883798415s" totalImagesPullingTime="4.383882034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:22.882856805 +0000 UTC m=+72.943332189" watchObservedRunningTime="2026-07-29 02:36:22.883798415 +0000 UTC m=+72.944273822" Jul 29 02:36:22.896331 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:22.896289 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2rsk8" podStartSLOduration=68.543474318 podStartE2EDuration="1m12.896275957s" totalImagesPullingTime="4.352801639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 02:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:22.896145996 +0000 UTC m=+72.956621381" watchObservedRunningTime="2026-07-29 02:36:22.896275957 +0000 UTC m=+72.956751341" Jul 29 02:36:28.467726 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:28.467700 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:36:30.794604 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:30.794577 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76c7df8f66-j5gz5" Jul 29 02:36:33.481047 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.480908 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" podUID="20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" containerName="registry" containerID="cri-o://3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac" gracePeriod=30 Jul 29 02:36:33.724839 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.724817 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:36:33.841194 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841143 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-installation-pull-secrets\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841194 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841179 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-ca-trust-extracted\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841340 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841198 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-bound-sa-token\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841340 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841340 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dms5k\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-kube-api-access-dms5k\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841340 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841311 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-certificates\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841340 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841334 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-trusted-ca\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841580 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-image-registry-private-configuration\") pod \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\" (UID: \"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce\") " Jul 29 02:36:33.841872 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 02:36:33.841872 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.841815 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 02:36:33.844070 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.844047 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:36:33.844164 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.844123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:36:33.844222 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.844196 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:36:33.844271 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.844250 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-kube-api-access-dms5k" (OuterVolumeSpecName: "kube-api-access-dms5k") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "kube-api-access-dms5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:36:33.844377 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.844361 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:36:33.849683 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.849663 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" (UID: "20eef5a0-f2e5-487c-8b39-57c60fe1c9ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:36:33.905129 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.905107 2579 generic.go:356] "Generic (PLEG): container finished" podID="20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" containerID="3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac" exitCode=0 Jul 29 02:36:33.905201 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.905140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" event={"ID":"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce","Type":"ContainerDied","Data":"3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac"} Jul 29 02:36:33.905201 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.905160 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" Jul 29 02:36:33.905201 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.905196 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c688544ff-lnvqd" event={"ID":"20eef5a0-f2e5-487c-8b39-57c60fe1c9ce","Type":"ContainerDied","Data":"3ea1689310c4189e48c900b76116f6e38f37b5160d79e49aeecba19a0fda1c60"} Jul 29 02:36:33.905290 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.905223 2579 scope.go:117] "RemoveContainer" containerID="3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac" Jul 29 02:36:33.912531 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.912511 2579 scope.go:117] "RemoveContainer" containerID="3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac" Jul 29 02:36:33.912819 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:36:33.912795 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac\": container with ID starting with 3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac not found: ID does not exist" containerID="3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac" Jul 29 02:36:33.912882 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.912830 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac"} err="failed to get container status \"3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac\": rpc error: code = NotFound desc = could not find container \"3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac\": container with ID starting with 3d1190cc4bd3c739faed13c35613d9818d6600852db86dc58842c0116723a7ac not found: ID does not exist" Jul 29 02:36:33.942319 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942298 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-installation-pull-secrets\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.942319 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942315 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-ca-trust-extracted\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.942442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942326 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-bound-sa-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.942442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942335 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.942442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942341 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dms5k\" (UniqueName: \"kubernetes.io/projected/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-kube-api-access-dms5k\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.942442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942349 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-registry-certificates\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.942442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942357 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-trusted-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.942442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.942366 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce-image-registry-private-configuration\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:36:33.952775 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.952750 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c688544ff-lnvqd"] Jul 29 02:36:33.963377 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:33.963359 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c688544ff-lnvqd"] Jul 29 02:36:34.450294 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:34.450274 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" path="/var/lib/kubelet/pods/20eef5a0-f2e5-487c-8b39-57c60fe1c9ce/volumes" Jul 29 02:36:42.989404 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:42.989370 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tgsvz"] Jul 29 02:36:42.989907 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:42.989669 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" containerName="registry" Jul 29 02:36:42.989907 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:42.989684 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" containerName="registry" Jul 29 02:36:42.989907 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:42.989759 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20eef5a0-f2e5-487c-8b39-57c60fe1c9ce" containerName="registry" Jul 29 02:36:43.161004 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.160976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.163816 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.163795 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 02:36:43.163975 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.163833 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 02:36:43.164094 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.164077 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 02:36:43.164148 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.164107 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 02:36:43.164542 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.164527 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 02:36:43.164620 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.164546 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 02:36:43.164620 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.164528 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8zmpg\"" Jul 29 02:36:43.300346 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300346 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-tls\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-root\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-sys\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300552 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8r8\" (UniqueName: \"kubernetes.io/projected/743bdc7a-7233-4a46-b461-18c5883e949f-kube-api-access-vr8r8\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300552 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300631 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-wtmp\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300631 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-textfile\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.300702 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.300651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/743bdc7a-7233-4a46-b461-18c5883e949f-metrics-client-ca\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.400837 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.400816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-wtmp\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.400928 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.400841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-textfile\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.400928 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.400859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/743bdc7a-7233-4a46-b461-18c5883e949f-metrics-client-ca\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.400928 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.400877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401096 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.400959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-tls\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401096 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-wtmp\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401096 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-root\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401096 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-sys\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401273 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8r8\" (UniqueName: \"kubernetes.io/projected/743bdc7a-7233-4a46-b461-18c5883e949f-kube-api-access-vr8r8\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401273 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401273 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-root\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401273 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-textfile\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401273 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/743bdc7a-7233-4a46-b461-18c5883e949f-sys\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401493 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.401550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.401505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/743bdc7a-7233-4a46-b461-18c5883e949f-metrics-client-ca\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.403159 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.403140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-tls\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.403249 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.403173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/743bdc7a-7233-4a46-b461-18c5883e949f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.410108 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.410080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8r8\" (UniqueName: \"kubernetes.io/projected/743bdc7a-7233-4a46-b461-18c5883e949f-kube-api-access-vr8r8\") pod \"node-exporter-tgsvz\" (UID: \"743bdc7a-7233-4a46-b461-18c5883e949f\") " pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.469741 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.469715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgsvz" Jul 29 02:36:43.518263 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:36:43.518240 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743bdc7a_7233_4a46_b461_18c5883e949f.slice/crio-3f221c708e7ca23c6c7bd52566045e9108ed8b80cfb9feafd7a4cb23ac2500cc WatchSource:0}: Error finding container 3f221c708e7ca23c6c7bd52566045e9108ed8b80cfb9feafd7a4cb23ac2500cc: Status 404 returned error can't find the container with id 3f221c708e7ca23c6c7bd52566045e9108ed8b80cfb9feafd7a4cb23ac2500cc Jul 29 02:36:43.941663 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:43.941639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgsvz" event={"ID":"743bdc7a-7233-4a46-b461-18c5883e949f","Type":"ContainerStarted","Data":"3f221c708e7ca23c6c7bd52566045e9108ed8b80cfb9feafd7a4cb23ac2500cc"} Jul 29 02:36:45.954727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:45.954692 2579 generic.go:356] "Generic (PLEG): container finished" podID="743bdc7a-7233-4a46-b461-18c5883e949f" containerID="9455abfde249e1afabb57b9d12241937803b44ffd61ca6e6dcfb47376698d0a0" exitCode=0 Jul 29 02:36:45.954727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:45.954732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgsvz" event={"ID":"743bdc7a-7233-4a46-b461-18c5883e949f","Type":"ContainerDied","Data":"9455abfde249e1afabb57b9d12241937803b44ffd61ca6e6dcfb47376698d0a0"} Jul 29 02:36:46.960069 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:46.960039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgsvz" event={"ID":"743bdc7a-7233-4a46-b461-18c5883e949f","Type":"ContainerStarted","Data":"07f8ca2a02d314813bea69ebeedfd45030116b4b14a124508e93ab7cd3384d44"} Jul 29 02:36:46.960411 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:46.960086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgsvz" event={"ID":"743bdc7a-7233-4a46-b461-18c5883e949f","Type":"ContainerStarted","Data":"70e94de7799d2d7c0fd34b6c2b3d839d1d41dd239c370d2e9db0f970e81137db"} Jul 29 02:36:46.976272 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:46.976233 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tgsvz" podStartSLOduration=3.510151458 podStartE2EDuration="4.976218734s" totalImagesPullingTime="1.391635033s" totalInitContainerRuntime="74.432243ms" isStatefulPod=true podCreationTimestamp="2026-07-29 02:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:36:46.974929544 +0000 UTC m=+97.035404941" watchObservedRunningTime="2026-07-29 02:36:46.976218734 +0000 UTC m=+97.036694118" Jul 29 02:36:53.874432 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:36:53.874405 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jz5m2" Jul 29 02:39:57.232615 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.232576 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q"] Jul 29 02:39:57.235198 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.235166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.237519 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.237473 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 02:39:57.237998 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.237952 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9nqt7\"" Jul 29 02:39:57.238124 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.237985 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 02:39:57.243174 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.243089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q"] Jul 29 02:39:57.318815 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.318772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkgp\" (UniqueName: \"kubernetes.io/projected/dfdd4671-9fcd-42c7-85f9-003da06edfb2-kube-api-access-xvkgp\") pod \"openshift-lws-operator-fd8ccff4c-twj6q\" (UID: \"dfdd4671-9fcd-42c7-85f9-003da06edfb2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.318815 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.318819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfdd4671-9fcd-42c7-85f9-003da06edfb2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-twj6q\" (UID: \"dfdd4671-9fcd-42c7-85f9-003da06edfb2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.419774 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.419733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkgp\" (UniqueName: \"kubernetes.io/projected/dfdd4671-9fcd-42c7-85f9-003da06edfb2-kube-api-access-xvkgp\") pod \"openshift-lws-operator-fd8ccff4c-twj6q\" (UID: \"dfdd4671-9fcd-42c7-85f9-003da06edfb2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.419774 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.419774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfdd4671-9fcd-42c7-85f9-003da06edfb2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-twj6q\" (UID: \"dfdd4671-9fcd-42c7-85f9-003da06edfb2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.420165 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.420150 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dfdd4671-9fcd-42c7-85f9-003da06edfb2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-twj6q\" (UID: \"dfdd4671-9fcd-42c7-85f9-003da06edfb2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.426716 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.426689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkgp\" (UniqueName: \"kubernetes.io/projected/dfdd4671-9fcd-42c7-85f9-003da06edfb2-kube-api-access-xvkgp\") pod \"openshift-lws-operator-fd8ccff4c-twj6q\" (UID: \"dfdd4671-9fcd-42c7-85f9-003da06edfb2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.545553 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.545466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" Jul 29 02:39:57.664980 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:57.664953 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q"] Jul 29 02:39:57.667795 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:39:57.667768 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfdd4671_9fcd_42c7_85f9_003da06edfb2.slice/crio-eab74d3400f5783c577574917fbb622277da3b2dcff108ce3e2b164076f6c6fd WatchSource:0}: Error finding container eab74d3400f5783c577574917fbb622277da3b2dcff108ce3e2b164076f6c6fd: Status 404 returned error can't find the container with id eab74d3400f5783c577574917fbb622277da3b2dcff108ce3e2b164076f6c6fd Jul 29 02:39:58.514143 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:39:58.514103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" event={"ID":"dfdd4671-9fcd-42c7-85f9-003da06edfb2","Type":"ContainerStarted","Data":"eab74d3400f5783c577574917fbb622277da3b2dcff108ce3e2b164076f6c6fd"} Jul 29 02:40:00.522834 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:00.522729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" event={"ID":"dfdd4671-9fcd-42c7-85f9-003da06edfb2","Type":"ContainerStarted","Data":"34d7f621a70b5c0f56a7df375b3458feae35fbf9ae76453da4bc0dc8d36e964d"} Jul 29 02:40:00.537699 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:00.537647 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-twj6q" podStartSLOduration=1.019260927 podStartE2EDuration="3.537629921s" totalImagesPullingTime="2.518368994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:00.536645892 +0000 UTC m=+290.597121278" watchObservedRunningTime="2026-07-29 02:40:00.537629921 +0000 UTC m=+290.598105308" Jul 29 02:40:10.296944 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:10.296911 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:40:10.300758 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:10.300734 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:40:10.303579 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:10.303558 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 02:40:17.206500 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.206462 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf"] Jul 29 02:40:17.212319 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.212302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.215922 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.215900 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 02:40:17.216054 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.215926 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 02:40:17.216502 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.216484 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 02:40:17.216657 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.216640 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 02:40:17.216657 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.216654 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wmb4h\"" Jul 29 02:40:17.245926 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.245900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf"] Jul 29 02:40:17.363676 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.363642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4187b9e7-bba0-4ff9-a987-0179b32924e0-manifests\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.363676 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.363688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khcdz\" (UniqueName: \"kubernetes.io/projected/4187b9e7-bba0-4ff9-a987-0179b32924e0-kube-api-access-khcdz\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.363900 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.363714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4187b9e7-bba0-4ff9-a987-0179b32924e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.363900 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.363757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4187b9e7-bba0-4ff9-a987-0179b32924e0-tmp\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.363900 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.363821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4187b9e7-bba0-4ff9-a987-0179b32924e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.464620 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.464530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4187b9e7-bba0-4ff9-a987-0179b32924e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.464620 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.464578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4187b9e7-bba0-4ff9-a987-0179b32924e0-manifests\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.464849 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.464699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khcdz\" (UniqueName: \"kubernetes.io/projected/4187b9e7-bba0-4ff9-a987-0179b32924e0-kube-api-access-khcdz\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.464849 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.464742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4187b9e7-bba0-4ff9-a987-0179b32924e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.464849 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.464774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4187b9e7-bba0-4ff9-a987-0179b32924e0-tmp\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.465004 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.464978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4187b9e7-bba0-4ff9-a987-0179b32924e0-manifests\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.467043 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.467007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4187b9e7-bba0-4ff9-a987-0179b32924e0-tmp\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.467148 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.467133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4187b9e7-bba0-4ff9-a987-0179b32924e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.467279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.467262 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4187b9e7-bba0-4ff9-a987-0179b32924e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.480958 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.480935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khcdz\" (UniqueName: \"kubernetes.io/projected/4187b9e7-bba0-4ff9-a987-0179b32924e0-kube-api-access-khcdz\") pod \"opendatahub-operator-controller-manager-5dbd45849d-857zf\" (UID: \"4187b9e7-bba0-4ff9-a987-0179b32924e0\") " pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.522111 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.522081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:17.650129 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.650094 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf"] Jul 29 02:40:17.653361 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:17.653326 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4187b9e7_bba0_4ff9_a987_0179b32924e0.slice/crio-82e79d0877c3308f9bcb38935dfcf92f53ac7c21b2af7f3d260743fa5193e942 WatchSource:0}: Error finding container 82e79d0877c3308f9bcb38935dfcf92f53ac7c21b2af7f3d260743fa5193e942: Status 404 returned error can't find the container with id 82e79d0877c3308f9bcb38935dfcf92f53ac7c21b2af7f3d260743fa5193e942 Jul 29 02:40:17.655040 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:17.655008 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 02:40:18.588808 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:18.588733 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" event={"ID":"4187b9e7-bba0-4ff9-a987-0179b32924e0","Type":"ContainerStarted","Data":"82e79d0877c3308f9bcb38935dfcf92f53ac7c21b2af7f3d260743fa5193e942"} Jul 29 02:40:21.601980 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:21.601934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" event={"ID":"4187b9e7-bba0-4ff9-a987-0179b32924e0","Type":"ContainerStarted","Data":"97aeccb917533168344a79f78fc976700d6585e4c8c236061327bddc57c05612"} Jul 29 02:40:22.609200 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:22.609162 2579 generic.go:356] "Generic (PLEG): container finished" podID="4187b9e7-bba0-4ff9-a987-0179b32924e0" containerID="97aeccb917533168344a79f78fc976700d6585e4c8c236061327bddc57c05612" exitCode=0 Jul 29 02:40:22.609625 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:22.609249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" event={"ID":"4187b9e7-bba0-4ff9-a987-0179b32924e0","Type":"ContainerDied","Data":"97aeccb917533168344a79f78fc976700d6585e4c8c236061327bddc57c05612"} Jul 29 02:40:23.614091 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:23.614055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" event={"ID":"4187b9e7-bba0-4ff9-a987-0179b32924e0","Type":"ContainerStarted","Data":"dcf8010b0b54fb3f5b02c0797f7cbac17a9fbfba4d7d8b06b80103befd97c8bc"} Jul 29 02:40:23.614489 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:23.614199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:23.631380 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:23.631316 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" podStartSLOduration=2.380037058 podStartE2EDuration="6.631299027s" totalImagesPullingTime="3.233548994s" totalInitContainerRuntime="1.017712975s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:23.630364572 +0000 UTC m=+313.690839977" watchObservedRunningTime="2026-07-29 02:40:23.631299027 +0000 UTC m=+313.691774413" Jul 29 02:40:34.620674 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:34.620636 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5dbd45849d-857zf" Jul 29 02:40:37.159782 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.159745 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2jldv"] Jul 29 02:40:37.162722 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.162705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.164678 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.164655 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 29 02:40:37.164678 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.164666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tzxfg\"" Jul 29 02:40:37.171108 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.171070 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2jldv"] Jul 29 02:40:37.222963 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.222930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8abdbca-5306-4318-acbc-bdbc6aef21a2-cert\") pod \"odh-model-controller-7998c67dc4-2jldv\" (UID: \"d8abdbca-5306-4318-acbc-bdbc6aef21a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.223129 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.222987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw84x\" (UniqueName: \"kubernetes.io/projected/d8abdbca-5306-4318-acbc-bdbc6aef21a2-kube-api-access-bw84x\") pod \"odh-model-controller-7998c67dc4-2jldv\" (UID: \"d8abdbca-5306-4318-acbc-bdbc6aef21a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.323366 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.323329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8abdbca-5306-4318-acbc-bdbc6aef21a2-cert\") pod \"odh-model-controller-7998c67dc4-2jldv\" (UID: \"d8abdbca-5306-4318-acbc-bdbc6aef21a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.323605 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.323413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw84x\" (UniqueName: \"kubernetes.io/projected/d8abdbca-5306-4318-acbc-bdbc6aef21a2-kube-api-access-bw84x\") pod \"odh-model-controller-7998c67dc4-2jldv\" (UID: \"d8abdbca-5306-4318-acbc-bdbc6aef21a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.323678 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:40:37.323663 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 29 02:40:37.323752 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:40:37.323732 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8abdbca-5306-4318-acbc-bdbc6aef21a2-cert podName:d8abdbca-5306-4318-acbc-bdbc6aef21a2 nodeName:}" failed. No retries permitted until 2026-07-29 02:40:37.823710843 +0000 UTC m=+327.884186224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d8abdbca-5306-4318-acbc-bdbc6aef21a2-cert") pod "odh-model-controller-7998c67dc4-2jldv" (UID: "d8abdbca-5306-4318-acbc-bdbc6aef21a2") : secret "odh-model-controller-webhook-cert" not found Jul 29 02:40:37.330992 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.330968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw84x\" (UniqueName: \"kubernetes.io/projected/d8abdbca-5306-4318-acbc-bdbc6aef21a2-kube-api-access-bw84x\") pod \"odh-model-controller-7998c67dc4-2jldv\" (UID: \"d8abdbca-5306-4318-acbc-bdbc6aef21a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.826996 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.826952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8abdbca-5306-4318-acbc-bdbc6aef21a2-cert\") pod \"odh-model-controller-7998c67dc4-2jldv\" (UID: \"d8abdbca-5306-4318-acbc-bdbc6aef21a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.829481 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.829459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8abdbca-5306-4318-acbc-bdbc6aef21a2-cert\") pod \"odh-model-controller-7998c67dc4-2jldv\" (UID: \"d8abdbca-5306-4318-acbc-bdbc6aef21a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:37.832755 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.832732 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q6tnz"] Jul 29 02:40:37.835695 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.835675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:37.837579 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.837562 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 02:40:37.837902 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.837885 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mtc8w\"" Jul 29 02:40:37.846261 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.846236 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q6tnz"] Jul 29 02:40:37.927590 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.927557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440221a6-cc0f-46a5-afe8-da54477362df-tls-certs\") pod \"model-serving-api-54d89bd5fc-q6tnz\" (UID: \"440221a6-cc0f-46a5-afe8-da54477362df\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:37.927771 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:37.927601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw729\" (UniqueName: \"kubernetes.io/projected/440221a6-cc0f-46a5-afe8-da54477362df-kube-api-access-nw729\") pod \"model-serving-api-54d89bd5fc-q6tnz\" (UID: \"440221a6-cc0f-46a5-afe8-da54477362df\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:38.028406 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.028366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440221a6-cc0f-46a5-afe8-da54477362df-tls-certs\") pod \"model-serving-api-54d89bd5fc-q6tnz\" (UID: \"440221a6-cc0f-46a5-afe8-da54477362df\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:38.028576 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.028424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw729\" (UniqueName: \"kubernetes.io/projected/440221a6-cc0f-46a5-afe8-da54477362df-kube-api-access-nw729\") pod \"model-serving-api-54d89bd5fc-q6tnz\" (UID: \"440221a6-cc0f-46a5-afe8-da54477362df\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:38.030829 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.030794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440221a6-cc0f-46a5-afe8-da54477362df-tls-certs\") pod \"model-serving-api-54d89bd5fc-q6tnz\" (UID: \"440221a6-cc0f-46a5-afe8-da54477362df\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:38.035241 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.035220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw729\" (UniqueName: \"kubernetes.io/projected/440221a6-cc0f-46a5-afe8-da54477362df-kube-api-access-nw729\") pod \"model-serving-api-54d89bd5fc-q6tnz\" (UID: \"440221a6-cc0f-46a5-afe8-da54477362df\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:38.073111 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.073079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:38.146363 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.146335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:38.198411 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.198376 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2jldv"] Jul 29 02:40:38.202344 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:38.202306 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8abdbca_5306_4318_acbc_bdbc6aef21a2.slice/crio-6b2fcacd043c5b5e7897312f804a1d3bba50a0f60656f45161ee43fe834189eb WatchSource:0}: Error finding container 6b2fcacd043c5b5e7897312f804a1d3bba50a0f60656f45161ee43fe834189eb: Status 404 returned error can't find the container with id 6b2fcacd043c5b5e7897312f804a1d3bba50a0f60656f45161ee43fe834189eb Jul 29 02:40:38.275518 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.275487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q6tnz"] Jul 29 02:40:38.278502 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:38.278479 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440221a6_cc0f_46a5_afe8_da54477362df.slice/crio-558dc5ed2ed561dea0f49073bd0e819c5a67199d71e89b47d7839ac5ae97ba56 WatchSource:0}: Error finding container 558dc5ed2ed561dea0f49073bd0e819c5a67199d71e89b47d7839ac5ae97ba56: Status 404 returned error can't find the container with id 558dc5ed2ed561dea0f49073bd0e819c5a67199d71e89b47d7839ac5ae97ba56 Jul 29 02:40:38.674147 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.674112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" event={"ID":"440221a6-cc0f-46a5-afe8-da54477362df","Type":"ContainerStarted","Data":"558dc5ed2ed561dea0f49073bd0e819c5a67199d71e89b47d7839ac5ae97ba56"} Jul 29 02:40:38.675235 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:38.675207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" event={"ID":"d8abdbca-5306-4318-acbc-bdbc6aef21a2","Type":"ContainerStarted","Data":"6b2fcacd043c5b5e7897312f804a1d3bba50a0f60656f45161ee43fe834189eb"} Jul 29 02:40:41.688784 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:41.688741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" event={"ID":"440221a6-cc0f-46a5-afe8-da54477362df","Type":"ContainerStarted","Data":"fd6b13a073d62682a71f16c9a8c31fb5ba7d4cb071342dbedaacce3d888379e7"} Jul 29 02:40:41.689271 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:41.688848 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:41.690445 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:41.690421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" event={"ID":"d8abdbca-5306-4318-acbc-bdbc6aef21a2","Type":"ContainerStarted","Data":"88bcd7154f5e2e8313d48ccbb045b5c6b9eaa54083de35246a327745a6365b3f"} Jul 29 02:40:41.690544 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:41.690476 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:41.704374 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:41.704328 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" podStartSLOduration=1.849575106 podStartE2EDuration="4.70431549s" totalImagesPullingTime="2.854740384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:41.703004871 +0000 UTC m=+331.763480250" watchObservedRunningTime="2026-07-29 02:40:41.70431549 +0000 UTC m=+331.764790899" Jul 29 02:40:41.716698 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:41.716649 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" podStartSLOduration=1.78242594 podStartE2EDuration="4.71663711s" totalImagesPullingTime="2.93421117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:41.715647724 +0000 UTC m=+331.776123122" watchObservedRunningTime="2026-07-29 02:40:41.71663711 +0000 UTC m=+331.777112495" Jul 29 02:40:42.272520 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.272486 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn"] Jul 29 02:40:42.275679 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.275660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.277601 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.277574 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 02:40:42.277720 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.277630 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 02:40:42.277720 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.277705 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-42r5m\"" Jul 29 02:40:42.277826 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.277708 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 02:40:42.277867 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.277824 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 02:40:42.284659 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.284639 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn"] Jul 29 02:40:42.364300 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.364262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ebef28c-7b32-4b43-a0de-4a197da67ef5-tmp\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.364477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.364343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef28c-7b32-4b43-a0de-4a197da67ef5-tls-certs\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.364477 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.364375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnp58\" (UniqueName: \"kubernetes.io/projected/6ebef28c-7b32-4b43-a0de-4a197da67ef5-kube-api-access-wnp58\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.464883 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.464850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ebef28c-7b32-4b43-a0de-4a197da67ef5-tmp\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.465084 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.464904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef28c-7b32-4b43-a0de-4a197da67ef5-tls-certs\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.465084 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.464926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnp58\" (UniqueName: \"kubernetes.io/projected/6ebef28c-7b32-4b43-a0de-4a197da67ef5-kube-api-access-wnp58\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.467153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.467129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ebef28c-7b32-4b43-a0de-4a197da67ef5-tmp\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.467476 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.467456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebef28c-7b32-4b43-a0de-4a197da67ef5-tls-certs\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.472072 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.472048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnp58\" (UniqueName: \"kubernetes.io/projected/6ebef28c-7b32-4b43-a0de-4a197da67ef5-kube-api-access-wnp58\") pod \"kube-auth-proxy-7cdc76d774-lmbzn\" (UID: \"6ebef28c-7b32-4b43-a0de-4a197da67ef5\") " pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.586198 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.586119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" Jul 29 02:40:42.727679 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:42.727647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn"] Jul 29 02:40:42.731058 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:42.731037 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebef28c_7b32_4b43_a0de_4a197da67ef5.slice/crio-417a3a3ca5d57a969788eb2cca860d727f24a7ac95e9837882e7eb5ed661c04c WatchSource:0}: Error finding container 417a3a3ca5d57a969788eb2cca860d727f24a7ac95e9837882e7eb5ed661c04c: Status 404 returned error can't find the container with id 417a3a3ca5d57a969788eb2cca860d727f24a7ac95e9837882e7eb5ed661c04c Jul 29 02:40:43.273799 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.273762 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p"] Jul 29 02:40:43.278134 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.278113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.280029 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.279998 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 02:40:43.280155 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.280035 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 02:40:43.280155 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.280040 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 02:40:43.280258 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.280246 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 02:40:43.280292 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.280284 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dv2wr\"" Jul 29 02:40:43.289803 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.289775 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p"] Jul 29 02:40:43.373460 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.373422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.373636 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.373496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.373636 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.373571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.373636 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.373621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2gr\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-kube-api-access-vn2gr\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.373812 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.373673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.373812 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.373731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.373812 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.373784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/998941a2-4cda-4300-b0cd-66434e9129ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475022 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.474974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475022 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.475027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.475061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2gr\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-kube-api-access-vn2gr\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.475085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.475114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.475132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/998941a2-4cda-4300-b0cd-66434e9129ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.475179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.475819 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.475795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.478103 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.478078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.478210 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.478078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.478302 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.478279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/998941a2-4cda-4300-b0cd-66434e9129ab-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.478454 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.478434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.484764 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.484742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.485069 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.485044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2gr\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-kube-api-access-vn2gr\") pod \"istiod-openshift-gateway-7b7995d5cd-dx75p\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.592917 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.592828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:43.701177 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.701139 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" event={"ID":"6ebef28c-7b32-4b43-a0de-4a197da67ef5","Type":"ContainerStarted","Data":"417a3a3ca5d57a969788eb2cca860d727f24a7ac95e9837882e7eb5ed661c04c"} Jul 29 02:40:43.743243 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:43.743180 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p"] Jul 29 02:40:43.745845 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:43.745817 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998941a2_4cda_4300_b0cd_66434e9129ab.slice/crio-3f37f35799449c2a5a6f427c551677dd4f6330c97d964c1ce630bbd26de73d88 WatchSource:0}: Error finding container 3f37f35799449c2a5a6f427c551677dd4f6330c97d964c1ce630bbd26de73d88: Status 404 returned error can't find the container with id 3f37f35799449c2a5a6f427c551677dd4f6330c97d964c1ce630bbd26de73d88 Jul 29 02:40:44.707861 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:44.707820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" event={"ID":"998941a2-4cda-4300-b0cd-66434e9129ab","Type":"ContainerStarted","Data":"3f37f35799449c2a5a6f427c551677dd4f6330c97d964c1ce630bbd26de73d88"} Jul 29 02:40:45.714211 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:45.714172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" event={"ID":"6ebef28c-7b32-4b43-a0de-4a197da67ef5","Type":"ContainerStarted","Data":"1587e412785fd0b0f507f3c861810c5dd41b4f27c8c7bdb6fb7ca02856f42c81"} Jul 29 02:40:45.737148 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:45.737089 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cdc76d774-lmbzn" podStartSLOduration=1.421340198 podStartE2EDuration="3.737072038s" totalImagesPullingTime="2.31573184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:45.735527582 +0000 UTC m=+335.796002968" watchObservedRunningTime="2026-07-29 02:40:45.737072038 +0000 UTC m=+335.797547422" Jul 29 02:40:46.489583 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.489548 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7trrb"] Jul 29 02:40:46.496567 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.496534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.498802 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.498638 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 29 02:40:46.498802 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.498638 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nbf2k\"" Jul 29 02:40:46.507649 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.505828 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7trrb"] Jul 29 02:40:46.507649 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.507594 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr"] Jul 29 02:40:46.511290 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.511268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.514037 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.513638 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gzs5l\"" Jul 29 02:40:46.514037 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.513879 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 29 02:40:46.518122 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.518066 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr"] Jul 29 02:40:46.602689 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.602653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebe6d1fa-9def-431b-8092-0ac85bdfe965-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b7rjr\" (UID: \"ebe6d1fa-9def-431b-8092-0ac85bdfe965\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.602689 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.602694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxjh\" (UniqueName: \"kubernetes.io/projected/8f72f594-e7cc-4154-a777-77ba264f82f6-kube-api-access-mqxjh\") pod \"kserve-controller-manager-d6f4b65b8-7trrb\" (UID: \"8f72f594-e7cc-4154-a777-77ba264f82f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.602945 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.602767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f72f594-e7cc-4154-a777-77ba264f82f6-cert\") pod \"kserve-controller-manager-d6f4b65b8-7trrb\" (UID: \"8f72f594-e7cc-4154-a777-77ba264f82f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.602945 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.602802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdkr\" (UniqueName: \"kubernetes.io/projected/ebe6d1fa-9def-431b-8092-0ac85bdfe965-kube-api-access-qcdkr\") pod \"llmisvc-controller-manager-777c6bd7d5-b7rjr\" (UID: \"ebe6d1fa-9def-431b-8092-0ac85bdfe965\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.703893 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.703858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f72f594-e7cc-4154-a777-77ba264f82f6-cert\") pod \"kserve-controller-manager-d6f4b65b8-7trrb\" (UID: \"8f72f594-e7cc-4154-a777-77ba264f82f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.704118 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.703921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdkr\" (UniqueName: \"kubernetes.io/projected/ebe6d1fa-9def-431b-8092-0ac85bdfe965-kube-api-access-qcdkr\") pod \"llmisvc-controller-manager-777c6bd7d5-b7rjr\" (UID: \"ebe6d1fa-9def-431b-8092-0ac85bdfe965\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.704118 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.703970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebe6d1fa-9def-431b-8092-0ac85bdfe965-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b7rjr\" (UID: \"ebe6d1fa-9def-431b-8092-0ac85bdfe965\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.704118 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.704001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxjh\" (UniqueName: \"kubernetes.io/projected/8f72f594-e7cc-4154-a777-77ba264f82f6-kube-api-access-mqxjh\") pod \"kserve-controller-manager-d6f4b65b8-7trrb\" (UID: \"8f72f594-e7cc-4154-a777-77ba264f82f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.706852 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.706818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f72f594-e7cc-4154-a777-77ba264f82f6-cert\") pod \"kserve-controller-manager-d6f4b65b8-7trrb\" (UID: \"8f72f594-e7cc-4154-a777-77ba264f82f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.707130 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.707109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebe6d1fa-9def-431b-8092-0ac85bdfe965-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b7rjr\" (UID: \"ebe6d1fa-9def-431b-8092-0ac85bdfe965\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.712098 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.712070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdkr\" (UniqueName: \"kubernetes.io/projected/ebe6d1fa-9def-431b-8092-0ac85bdfe965-kube-api-access-qcdkr\") pod \"llmisvc-controller-manager-777c6bd7d5-b7rjr\" (UID: \"ebe6d1fa-9def-431b-8092-0ac85bdfe965\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.712294 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.712272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxjh\" (UniqueName: \"kubernetes.io/projected/8f72f594-e7cc-4154-a777-77ba264f82f6-kube-api-access-mqxjh\") pod \"kserve-controller-manager-d6f4b65b8-7trrb\" (UID: \"8f72f594-e7cc-4154-a777-77ba264f82f6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.810113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.810045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:46.827168 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.827147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:46.871589 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.871534 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:40:46.871697 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.871627 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:40:46.962706 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:46.962567 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7trrb"] Jul 29 02:40:46.968228 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:46.968180 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f72f594_e7cc_4154_a777_77ba264f82f6.slice/crio-5a1269cae6b7186a7be120080bef211d14adcb82b9b540ec787efe1116bba717 WatchSource:0}: Error finding container 5a1269cae6b7186a7be120080bef211d14adcb82b9b540ec787efe1116bba717: Status 404 returned error can't find the container with id 5a1269cae6b7186a7be120080bef211d14adcb82b9b540ec787efe1116bba717 Jul 29 02:40:47.013043 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:47.012995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr"] Jul 29 02:40:47.016282 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:47.016241 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podebe6d1fa_9def_431b_8092_0ac85bdfe965.slice/crio-78228bf2859dff9e06934fc49e8b707128bde72c128b07827fba4aa41e4486dc WatchSource:0}: Error finding container 78228bf2859dff9e06934fc49e8b707128bde72c128b07827fba4aa41e4486dc: Status 404 returned error can't find the container with id 78228bf2859dff9e06934fc49e8b707128bde72c128b07827fba4aa41e4486dc Jul 29 02:40:47.725709 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:47.725641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" event={"ID":"8f72f594-e7cc-4154-a777-77ba264f82f6","Type":"ContainerStarted","Data":"5a1269cae6b7186a7be120080bef211d14adcb82b9b540ec787efe1116bba717"} Jul 29 02:40:47.727171 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:47.727141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" event={"ID":"ebe6d1fa-9def-431b-8092-0ac85bdfe965","Type":"ContainerStarted","Data":"78228bf2859dff9e06934fc49e8b707128bde72c128b07827fba4aa41e4486dc"} Jul 29 02:40:47.729194 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:47.729159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" event={"ID":"998941a2-4cda-4300-b0cd-66434e9129ab","Type":"ContainerStarted","Data":"ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c"} Jul 29 02:40:47.729394 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:47.729376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:47.731494 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:47.731471 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-dx75p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 02:40:47.731599 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:47.731519 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" podUID="998941a2-4cda-4300-b0cd-66434e9129ab" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 02:40:48.734673 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:48.734637 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-dx75p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 02:40:48.735171 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:48.734696 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" podUID="998941a2-4cda-4300-b0cd-66434e9129ab" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 02:40:49.096104 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.095997 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" podStartSLOduration=2.972704981 podStartE2EDuration="6.095971083s" totalImagesPullingTime="3.123266102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:47.747446765 +0000 UTC m=+337.807922151" watchObservedRunningTime="2026-07-29 02:40:49.095971083 +0000 UTC m=+339.156446472" Jul 29 02:40:49.097267 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.097235 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb"] Jul 29 02:40:49.101351 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.100818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.102907 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.102826 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xpsm9\"" Jul 29 02:40:49.114034 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.112763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb"] Jul 29 02:40:49.229992 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.229960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5sgt\" (UniqueName: \"kubernetes.io/projected/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-kube-api-access-n5sgt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.230384 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.230311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331099 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5sgt\" (UniqueName: \"kubernetes.io/projected/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-kube-api-access-n5sgt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331295 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331295 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331295 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331471 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331471 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331471 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331471 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.331471 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.331439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.332353 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.332318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.332478 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.332433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.332478 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.332469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.332595 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.332534 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.332924 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.332905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.334550 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.334509 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.334802 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.334783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.340543 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.340522 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5sgt\" (UniqueName: \"kubernetes.io/projected/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-kube-api-access-n5sgt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.340790 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.340767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bd5a2c-bde0-4eb9-9308-c69a6eae8357-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb\" (UID: \"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.417951 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.417864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:49.956430 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:49.956395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb"] Jul 29 02:40:49.960220 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:40:49.960194 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bd5a2c_bde0_4eb9_9308_c69a6eae8357.slice/crio-eae2a4a7e1651be013cdf9cd5944325040f6f226c196ae10bd6781cf52048dc7 WatchSource:0}: Error finding container eae2a4a7e1651be013cdf9cd5944325040f6f226c196ae10bd6781cf52048dc7: Status 404 returned error can't find the container with id eae2a4a7e1651be013cdf9cd5944325040f6f226c196ae10bd6781cf52048dc7 Jul 29 02:40:50.748258 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:50.748220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" event={"ID":"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357","Type":"ContainerStarted","Data":"eae2a4a7e1651be013cdf9cd5944325040f6f226c196ae10bd6781cf52048dc7"} Jul 29 02:40:50.749666 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:50.749637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" event={"ID":"8f72f594-e7cc-4154-a777-77ba264f82f6","Type":"ContainerStarted","Data":"afcc69b519862358a26533c92f8e5ec805a09407508d4a6f0fd556f141b26da6"} Jul 29 02:40:50.749784 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:50.749712 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:40:50.751252 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:50.751228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" event={"ID":"ebe6d1fa-9def-431b-8092-0ac85bdfe965","Type":"ContainerStarted","Data":"4dfca0911c121cf4adedb52ad8a7b6d1dcfa50ea3dacd3d776254f86059d3991"} Jul 29 02:40:50.751389 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:50.751376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:40:50.762981 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:50.762933 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" podStartSLOduration=1.906291838 podStartE2EDuration="4.762919395s" totalImagesPullingTime="2.856627557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:50.762742771 +0000 UTC m=+340.823218156" watchObservedRunningTime="2026-07-29 02:40:50.762919395 +0000 UTC m=+340.823394782" Jul 29 02:40:50.775518 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:50.775465 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" podStartSLOduration=1.9561033079999999 podStartE2EDuration="4.775447858s" totalImagesPullingTime="2.81934455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:50.775105132 +0000 UTC m=+340.835580515" watchObservedRunningTime="2026-07-29 02:40:50.775447858 +0000 UTC m=+340.835923243" Jul 29 02:40:51.734947 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:51.734912 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:52.018205 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.017581 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p"] Jul 29 02:40:52.018205 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.017841 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" podUID="998941a2-4cda-4300-b0cd-66434e9129ab" containerName="discovery" containerID="cri-o://ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c" gracePeriod=30 Jul 29 02:40:52.279958 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.279899 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:52.357877 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.357803 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-istio-token\") pod \"998941a2-4cda-4300-b0cd-66434e9129ab\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " Jul 29 02:40:52.357877 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.357869 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-ca-configmap\") pod \"998941a2-4cda-4300-b0cd-66434e9129ab\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " Jul 29 02:40:52.358157 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.357913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn2gr\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-kube-api-access-vn2gr\") pod \"998941a2-4cda-4300-b0cd-66434e9129ab\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " Jul 29 02:40:52.358157 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.357946 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-dns-cert\") pod \"998941a2-4cda-4300-b0cd-66434e9129ab\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " Jul 29 02:40:52.358157 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.357982 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-kubeconfig\") pod \"998941a2-4cda-4300-b0cd-66434e9129ab\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " Jul 29 02:40:52.358157 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.358039 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/998941a2-4cda-4300-b0cd-66434e9129ab-local-certs\") pod \"998941a2-4cda-4300-b0cd-66434e9129ab\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " Jul 29 02:40:52.358157 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.358090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-cacerts\") pod \"998941a2-4cda-4300-b0cd-66434e9129ab\" (UID: \"998941a2-4cda-4300-b0cd-66434e9129ab\") " Jul 29 02:40:52.358384 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.358300 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "998941a2-4cda-4300-b0cd-66434e9129ab" (UID: "998941a2-4cda-4300-b0cd-66434e9129ab"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 02:40:52.360856 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.360820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "998941a2-4cda-4300-b0cd-66434e9129ab" (UID: "998941a2-4cda-4300-b0cd-66434e9129ab"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:40:52.361067 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.361036 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-cacerts" (OuterVolumeSpecName: "cacerts") pod "998941a2-4cda-4300-b0cd-66434e9129ab" (UID: "998941a2-4cda-4300-b0cd-66434e9129ab"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:40:52.361413 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.361395 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "998941a2-4cda-4300-b0cd-66434e9129ab" (UID: "998941a2-4cda-4300-b0cd-66434e9129ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:40:52.361513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.361494 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998941a2-4cda-4300-b0cd-66434e9129ab-local-certs" (OuterVolumeSpecName: "local-certs") pod "998941a2-4cda-4300-b0cd-66434e9129ab" (UID: "998941a2-4cda-4300-b0cd-66434e9129ab"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:40:52.361567 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.361515 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-kube-api-access-vn2gr" (OuterVolumeSpecName: "kube-api-access-vn2gr") pod "998941a2-4cda-4300-b0cd-66434e9129ab" (UID: "998941a2-4cda-4300-b0cd-66434e9129ab"). InnerVolumeSpecName "kube-api-access-vn2gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:40:52.361996 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.361971 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "998941a2-4cda-4300-b0cd-66434e9129ab" (UID: "998941a2-4cda-4300-b0cd-66434e9129ab"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:40:52.458827 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.458792 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn2gr\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-kube-api-access-vn2gr\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:40:52.458827 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.458828 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-dns-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:40:52.459075 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.458844 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-istio-kubeconfig\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:40:52.459075 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.458858 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/998941a2-4cda-4300-b0cd-66434e9129ab-local-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:40:52.459075 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.458874 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/998941a2-4cda-4300-b0cd-66434e9129ab-cacerts\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:40:52.459075 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.458885 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998941a2-4cda-4300-b0cd-66434e9129ab-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:40:52.459075 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.458897 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/998941a2-4cda-4300-b0cd-66434e9129ab-istio-csr-ca-configmap\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:40:52.697594 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.697536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2jldv" Jul 29 02:40:52.700986 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.700963 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-q6tnz" Jul 29 02:40:52.764075 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.764038 2579 generic.go:356] "Generic (PLEG): container finished" podID="998941a2-4cda-4300-b0cd-66434e9129ab" containerID="ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c" exitCode=0 Jul 29 02:40:52.764575 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.764122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" event={"ID":"998941a2-4cda-4300-b0cd-66434e9129ab","Type":"ContainerDied","Data":"ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c"} Jul 29 02:40:52.764575 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.764172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" event={"ID":"998941a2-4cda-4300-b0cd-66434e9129ab","Type":"ContainerDied","Data":"3f37f35799449c2a5a6f427c551677dd4f6330c97d964c1ce630bbd26de73d88"} Jul 29 02:40:52.764575 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.764177 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p" Jul 29 02:40:52.764575 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.764199 2579 scope.go:117] "RemoveContainer" containerID="ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c" Jul 29 02:40:52.780799 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.780754 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p"] Jul 29 02:40:52.784818 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:52.784791 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dx75p"] Jul 29 02:40:53.680961 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:53.680793 2579 scope.go:117] "RemoveContainer" containerID="ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c" Jul 29 02:40:53.681273 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:40:53.681246 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c\": container with ID starting with ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c not found: ID does not exist" containerID="ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c" Jul 29 02:40:53.681362 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:53.681285 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c"} err="failed to get container status \"ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c\": rpc error: code = NotFound desc = could not find container \"ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c\": container with ID starting with ead1de840907cf6e72528cfae662f479269696b44833349d19c4b02fb5d34d4c not found: ID does not exist" Jul 29 02:40:53.738736 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:53.738704 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:40:53.738855 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:53.738782 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:40:53.738855 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:53.738813 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:40:54.451679 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:54.451640 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998941a2-4cda-4300-b0cd-66434e9129ab" path="/var/lib/kubelet/pods/998941a2-4cda-4300-b0cd-66434e9129ab/volumes" Jul 29 02:40:54.774761 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:54.774669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" event={"ID":"f3bd5a2c-bde0-4eb9-9308-c69a6eae8357","Type":"ContainerStarted","Data":"1425f94a20f9208b8a6c14f8d1e558cc9e7aac6305f35455fd42e4a06591973b"} Jul 29 02:40:54.791596 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:54.791532 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" podStartSLOduration=2.015858847 podStartE2EDuration="5.791513352s" totalImagesPullingTime="3.775654505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:40:54.790738855 +0000 UTC m=+344.851214238" watchObservedRunningTime="2026-07-29 02:40:54.791513352 +0000 UTC m=+344.851988738" Jul 29 02:40:55.418078 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:55.418035 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:55.423975 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:55.423944 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:55.778621 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:55.778589 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:40:55.779627 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:40:55.779609 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb" Jul 29 02:41:21.758776 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:21.758746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b7rjr" Jul 29 02:41:21.761781 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:21.761760 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7trrb" Jul 29 02:41:38.108295 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.108259 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gwwl5"] Jul 29 02:41:38.108735 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.108547 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998941a2-4cda-4300-b0cd-66434e9129ab" containerName="discovery" Jul 29 02:41:38.108735 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.108557 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="998941a2-4cda-4300-b0cd-66434e9129ab" containerName="discovery" Jul 29 02:41:38.108735 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.108601 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="998941a2-4cda-4300-b0cd-66434e9129ab" containerName="discovery" Jul 29 02:41:38.111152 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.111134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" Jul 29 02:41:38.113427 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.113403 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 02:41:38.113561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.113411 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-gbb5t\"" Jul 29 02:41:38.113945 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.113913 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 02:41:38.123442 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.123416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gwwl5"] Jul 29 02:41:38.228428 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.228387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5g6p\" (UniqueName: \"kubernetes.io/projected/91f47dcc-c2c9-4276-9906-e94651483e3f-kube-api-access-c5g6p\") pod \"kuadrant-operator-catalog-gwwl5\" (UID: \"91f47dcc-c2c9-4276-9906-e94651483e3f\") " pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" Jul 29 02:41:38.329474 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.329437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5g6p\" (UniqueName: \"kubernetes.io/projected/91f47dcc-c2c9-4276-9906-e94651483e3f-kube-api-access-c5g6p\") pod \"kuadrant-operator-catalog-gwwl5\" (UID: \"91f47dcc-c2c9-4276-9906-e94651483e3f\") " pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" Jul 29 02:41:38.336669 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.336644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5g6p\" (UniqueName: \"kubernetes.io/projected/91f47dcc-c2c9-4276-9906-e94651483e3f-kube-api-access-c5g6p\") pod \"kuadrant-operator-catalog-gwwl5\" (UID: \"91f47dcc-c2c9-4276-9906-e94651483e3f\") " pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" Jul 29 02:41:38.423572 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.423482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" Jul 29 02:41:38.475374 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.475346 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gwwl5"] Jul 29 02:41:38.545845 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.545809 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gwwl5"] Jul 29 02:41:38.549073 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:41:38.549050 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f47dcc_c2c9_4276_9906_e94651483e3f.slice/crio-6b49d71dd833e7f98a877cd9faaaac31b250ce09bdb7563e79cb3bec87028870 WatchSource:0}: Error finding container 6b49d71dd833e7f98a877cd9faaaac31b250ce09bdb7563e79cb3bec87028870: Status 404 returned error can't find the container with id 6b49d71dd833e7f98a877cd9faaaac31b250ce09bdb7563e79cb3bec87028870 Jul 29 02:41:38.684815 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.684787 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b9kw"] Jul 29 02:41:38.688668 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.688653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:38.695189 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.695161 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b9kw"] Jul 29 02:41:38.733735 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.733687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvd7\" (UniqueName: \"kubernetes.io/projected/790610a4-7db1-420f-b481-a41d41a828f0-kube-api-access-qvvd7\") pod \"kuadrant-operator-catalog-4b9kw\" (UID: \"790610a4-7db1-420f-b481-a41d41a828f0\") " pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:38.834494 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.834458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvd7\" (UniqueName: \"kubernetes.io/projected/790610a4-7db1-420f-b481-a41d41a828f0-kube-api-access-qvvd7\") pod \"kuadrant-operator-catalog-4b9kw\" (UID: \"790610a4-7db1-420f-b481-a41d41a828f0\") " pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:38.843280 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.843254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvd7\" (UniqueName: \"kubernetes.io/projected/790610a4-7db1-420f-b481-a41d41a828f0-kube-api-access-qvvd7\") pod \"kuadrant-operator-catalog-4b9kw\" (UID: \"790610a4-7db1-420f-b481-a41d41a828f0\") " pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:38.951202 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.951127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" event={"ID":"91f47dcc-c2c9-4276-9906-e94651483e3f","Type":"ContainerStarted","Data":"6b49d71dd833e7f98a877cd9faaaac31b250ce09bdb7563e79cb3bec87028870"} Jul 29 02:41:38.998428 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:38.998400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:39.134432 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:39.134404 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b9kw"] Jul 29 02:41:39.173393 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:41:39.173362 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790610a4_7db1_420f_b481_a41d41a828f0.slice/crio-7e1755607dc69358211dc0e5265acf58e421822c3888fdfadce978f1e1376eb3 WatchSource:0}: Error finding container 7e1755607dc69358211dc0e5265acf58e421822c3888fdfadce978f1e1376eb3: Status 404 returned error can't find the container with id 7e1755607dc69358211dc0e5265acf58e421822c3888fdfadce978f1e1376eb3 Jul 29 02:41:39.956571 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:39.956539 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" event={"ID":"790610a4-7db1-420f-b481-a41d41a828f0","Type":"ContainerStarted","Data":"7e1755607dc69358211dc0e5265acf58e421822c3888fdfadce978f1e1376eb3"} Jul 29 02:41:40.961669 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:40.961627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" event={"ID":"91f47dcc-c2c9-4276-9906-e94651483e3f","Type":"ContainerStarted","Data":"fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9"} Jul 29 02:41:40.961669 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:40.961662 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" podUID="91f47dcc-c2c9-4276-9906-e94651483e3f" containerName="registry-server" containerID="cri-o://fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9" gracePeriod=2 Jul 29 02:41:40.963187 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:40.963162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" event={"ID":"790610a4-7db1-420f-b481-a41d41a828f0","Type":"ContainerStarted","Data":"6d82167d5113266bf60d8c97f58ddba68a064f9ed0467207a8e7118b855eb048"} Jul 29 02:41:40.975397 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:40.975348 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" podStartSLOduration=1.089579871 podStartE2EDuration="2.975331947s" totalImagesPullingTime="1.885752076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:41:40.974120243 +0000 UTC m=+391.034595618" watchObservedRunningTime="2026-07-29 02:41:40.975331947 +0000 UTC m=+391.035807351" Jul 29 02:41:40.988163 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:40.988116 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" podStartSLOduration=1.7255111730000001 podStartE2EDuration="2.988102375s" totalImagesPullingTime="1.262591202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:41:40.986429212 +0000 UTC m=+391.046904596" watchObservedRunningTime="2026-07-29 02:41:40.988102375 +0000 UTC m=+391.048577810" Jul 29 02:41:41.205921 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.205895 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" Jul 29 02:41:41.256257 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.256169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5g6p\" (UniqueName: \"kubernetes.io/projected/91f47dcc-c2c9-4276-9906-e94651483e3f-kube-api-access-c5g6p\") pod \"91f47dcc-c2c9-4276-9906-e94651483e3f\" (UID: \"91f47dcc-c2c9-4276-9906-e94651483e3f\") " Jul 29 02:41:41.258321 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.258297 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f47dcc-c2c9-4276-9906-e94651483e3f-kube-api-access-c5g6p" (OuterVolumeSpecName: "kube-api-access-c5g6p") pod "91f47dcc-c2c9-4276-9906-e94651483e3f" (UID: "91f47dcc-c2c9-4276-9906-e94651483e3f"). InnerVolumeSpecName "kube-api-access-c5g6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:41:41.356893 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.356851 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5g6p\" (UniqueName: \"kubernetes.io/projected/91f47dcc-c2c9-4276-9906-e94651483e3f-kube-api-access-c5g6p\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:41:41.968043 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.967983 2579 generic.go:356] "Generic (PLEG): container finished" podID="91f47dcc-c2c9-4276-9906-e94651483e3f" containerID="fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9" exitCode=0 Jul 29 02:41:41.968043 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.968044 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" Jul 29 02:41:41.968549 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.968057 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" event={"ID":"91f47dcc-c2c9-4276-9906-e94651483e3f","Type":"ContainerDied","Data":"fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9"} Jul 29 02:41:41.968549 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.968133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gwwl5" event={"ID":"91f47dcc-c2c9-4276-9906-e94651483e3f","Type":"ContainerDied","Data":"6b49d71dd833e7f98a877cd9faaaac31b250ce09bdb7563e79cb3bec87028870"} Jul 29 02:41:41.968549 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.968167 2579 scope.go:117] "RemoveContainer" containerID="fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9" Jul 29 02:41:41.977717 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.977700 2579 scope.go:117] "RemoveContainer" containerID="fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9" Jul 29 02:41:41.977984 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:41:41.977966 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9\": container with ID starting with fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9 not found: ID does not exist" containerID="fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9" Jul 29 02:41:41.978058 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.977995 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9"} err="failed to get container status \"fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9\": rpc error: code = NotFound desc = could not find container \"fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9\": container with ID starting with fbddfa3a2d17f83c563e6f5b776cd6eb7751a125ca3752b1241ea97448c126f9 not found: ID does not exist" Jul 29 02:41:41.988517 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.988485 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gwwl5"] Jul 29 02:41:41.992392 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:41.992360 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gwwl5"] Jul 29 02:41:42.451756 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:42.451726 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f47dcc-c2c9-4276-9906-e94651483e3f" path="/var/lib/kubelet/pods/91f47dcc-c2c9-4276-9906-e94651483e3f/volumes" Jul 29 02:41:48.998772 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:48.998738 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:48.999259 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:48.998787 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:49.019856 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:49.019831 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:41:50.022515 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:41:50.022486 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4b9kw" Jul 29 02:42:10.779958 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.779922 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6"] Jul 29 02:42:10.780359 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.780234 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f47dcc-c2c9-4276-9906-e94651483e3f" containerName="registry-server" Jul 29 02:42:10.780359 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.780245 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f47dcc-c2c9-4276-9906-e94651483e3f" containerName="registry-server" Jul 29 02:42:10.780359 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.780296 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f47dcc-c2c9-4276-9906-e94651483e3f" containerName="registry-server" Jul 29 02:42:10.784215 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.784196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" Jul 29 02:42:10.786630 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.786609 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8hmsp\"" Jul 29 02:42:10.793126 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.793035 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6"] Jul 29 02:42:10.908587 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:10.908546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tn9k\" (UniqueName: \"kubernetes.io/projected/b7206433-81e2-4331-9650-33570a02e3e8-kube-api-access-9tn9k\") pod \"authorino-operator-75cc8d5cb7-dc4c6\" (UID: \"b7206433-81e2-4331-9650-33570a02e3e8\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" Jul 29 02:42:11.009469 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:11.009429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tn9k\" (UniqueName: \"kubernetes.io/projected/b7206433-81e2-4331-9650-33570a02e3e8-kube-api-access-9tn9k\") pod \"authorino-operator-75cc8d5cb7-dc4c6\" (UID: \"b7206433-81e2-4331-9650-33570a02e3e8\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" Jul 29 02:42:11.036625 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:11.036547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tn9k\" (UniqueName: \"kubernetes.io/projected/b7206433-81e2-4331-9650-33570a02e3e8-kube-api-access-9tn9k\") pod \"authorino-operator-75cc8d5cb7-dc4c6\" (UID: \"b7206433-81e2-4331-9650-33570a02e3e8\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" Jul 29 02:42:11.095092 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:11.095065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" Jul 29 02:42:11.224928 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:11.224898 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6"] Jul 29 02:42:11.227801 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:42:11.227772 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7206433_81e2_4331_9650_33570a02e3e8.slice/crio-5344a32fecbcff4975a4610e4e9aea4dae9d347e1ab3450e3dafb7e77f7359e5 WatchSource:0}: Error finding container 5344a32fecbcff4975a4610e4e9aea4dae9d347e1ab3450e3dafb7e77f7359e5: Status 404 returned error can't find the container with id 5344a32fecbcff4975a4610e4e9aea4dae9d347e1ab3450e3dafb7e77f7359e5 Jul 29 02:42:12.093701 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:12.093665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" event={"ID":"b7206433-81e2-4331-9650-33570a02e3e8","Type":"ContainerStarted","Data":"5344a32fecbcff4975a4610e4e9aea4dae9d347e1ab3450e3dafb7e77f7359e5"} Jul 29 02:42:13.765037 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:13.764988 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl"] Jul 29 02:42:13.768870 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:13.768851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:13.770895 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:13.770872 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7zvhw\"" Jul 29 02:42:13.778327 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:13.778300 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl"] Jul 29 02:42:13.829174 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:13.829124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zc6d\" (UniqueName: \"kubernetes.io/projected/5737718d-e914-4f3f-962f-fc38828a42f3-kube-api-access-2zc6d\") pod \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" (UID: \"5737718d-e914-4f3f-962f-fc38828a42f3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:13.930261 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:13.930222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zc6d\" (UniqueName: \"kubernetes.io/projected/5737718d-e914-4f3f-962f-fc38828a42f3-kube-api-access-2zc6d\") pod \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" (UID: \"5737718d-e914-4f3f-962f-fc38828a42f3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:13.948886 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:13.948858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zc6d\" (UniqueName: \"kubernetes.io/projected/5737718d-e914-4f3f-962f-fc38828a42f3-kube-api-access-2zc6d\") pod \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" (UID: \"5737718d-e914-4f3f-962f-fc38828a42f3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:14.080128 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:14.080041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:14.104869 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:14.104833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" event={"ID":"b7206433-81e2-4331-9650-33570a02e3e8","Type":"ContainerStarted","Data":"8bd3ed12926e5466bffb7a5a1f2c4ad761e7f271ad89581198026bf2e4fea0a5"} Jul 29 02:42:14.105074 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:14.104964 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" Jul 29 02:42:14.121458 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:14.121248 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" podStartSLOduration=2.023707178 podStartE2EDuration="4.121228143s" totalImagesPullingTime="2.097520965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:42:14.120619509 +0000 UTC m=+424.181094906" watchObservedRunningTime="2026-07-29 02:42:14.121228143 +0000 UTC m=+424.181703527" Jul 29 02:42:14.207729 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:14.207696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl"] Jul 29 02:42:14.210662 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:42:14.210631 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5737718d_e914_4f3f_962f_fc38828a42f3.slice/crio-e7ca62c782202d33876f4853a92c7f2f2a288a2dfa24fd4bcfdbb133da971721 WatchSource:0}: Error finding container e7ca62c782202d33876f4853a92c7f2f2a288a2dfa24fd4bcfdbb133da971721: Status 404 returned error can't find the container with id e7ca62c782202d33876f4853a92c7f2f2a288a2dfa24fd4bcfdbb133da971721 Jul 29 02:42:15.111175 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:15.111140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" event={"ID":"5737718d-e914-4f3f-962f-fc38828a42f3","Type":"ContainerStarted","Data":"e7ca62c782202d33876f4853a92c7f2f2a288a2dfa24fd4bcfdbb133da971721"} Jul 29 02:42:16.121857 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:16.121751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" event={"ID":"5737718d-e914-4f3f-962f-fc38828a42f3","Type":"ContainerStarted","Data":"1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c"} Jul 29 02:42:16.122271 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:16.121866 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:16.137820 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:16.137769 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" podStartSLOduration=1.537747157 podStartE2EDuration="3.137752041s" totalImagesPullingTime="1.600004884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:42:16.136578795 +0000 UTC m=+426.197054370" watchObservedRunningTime="2026-07-29 02:42:16.137752041 +0000 UTC m=+426.198227407" Jul 29 02:42:17.138995 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.138959 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv"] Jul 29 02:42:17.142331 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.142315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" Jul 29 02:42:17.144859 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.144837 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 29 02:42:17.144960 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.144837 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ccxj4\"" Jul 29 02:42:17.153361 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.153337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2hr\" (UniqueName: \"kubernetes.io/projected/5a1ff45b-4a0b-441a-b112-411303f26976-kube-api-access-pq2hr\") pod \"dns-operator-controller-manager-864dd78c4c-qqpwv\" (UID: \"5a1ff45b-4a0b-441a-b112-411303f26976\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" Jul 29 02:42:17.156074 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.156052 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv"] Jul 29 02:42:17.254261 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.254220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2hr\" (UniqueName: \"kubernetes.io/projected/5a1ff45b-4a0b-441a-b112-411303f26976-kube-api-access-pq2hr\") pod \"dns-operator-controller-manager-864dd78c4c-qqpwv\" (UID: \"5a1ff45b-4a0b-441a-b112-411303f26976\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" Jul 29 02:42:17.264263 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.264228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2hr\" (UniqueName: \"kubernetes.io/projected/5a1ff45b-4a0b-441a-b112-411303f26976-kube-api-access-pq2hr\") pod \"dns-operator-controller-manager-864dd78c4c-qqpwv\" (UID: \"5a1ff45b-4a0b-441a-b112-411303f26976\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" Jul 29 02:42:17.452307 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.452274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" Jul 29 02:42:17.580921 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:17.580890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv"] Jul 29 02:42:17.584060 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:42:17.584028 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1ff45b_4a0b_441a_b112_411303f26976.slice/crio-3d2cd938bf1a59871fd7cf3e50eac4e78b5eb63edc5d2b16d93b572da8e82e7e WatchSource:0}: Error finding container 3d2cd938bf1a59871fd7cf3e50eac4e78b5eb63edc5d2b16d93b572da8e82e7e: Status 404 returned error can't find the container with id 3d2cd938bf1a59871fd7cf3e50eac4e78b5eb63edc5d2b16d93b572da8e82e7e Jul 29 02:42:18.131531 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:18.131494 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" event={"ID":"5a1ff45b-4a0b-441a-b112-411303f26976","Type":"ContainerStarted","Data":"3d2cd938bf1a59871fd7cf3e50eac4e78b5eb63edc5d2b16d93b572da8e82e7e"} Jul 29 02:42:20.143979 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:20.143945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" event={"ID":"5a1ff45b-4a0b-441a-b112-411303f26976","Type":"ContainerStarted","Data":"e37950699981f2104a1c9a6a7eb326e51735b18afc532707d44d0f4ff597563b"} Jul 29 02:42:20.144470 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:20.144059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" Jul 29 02:42:20.161051 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:20.160871 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" podStartSLOduration=1.514623348 podStartE2EDuration="3.160852925s" totalImagesPullingTime="1.646229577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:42:20.160160918 +0000 UTC m=+430.220636300" watchObservedRunningTime="2026-07-29 02:42:20.160852925 +0000 UTC m=+430.221328313" Jul 29 02:42:22.216855 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.216820 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh"] Jul 29 02:42:22.220241 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.220220 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.223237 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.223215 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nq5jt\"" Jul 29 02:42:22.234556 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.234531 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh"] Jul 29 02:42:22.294033 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.293977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq77l\" (UniqueName: \"kubernetes.io/projected/f1491277-0af3-4314-910c-ccbf15867809-kube-api-access-zq77l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.294200 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.294088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1491277-0af3-4314-910c-ccbf15867809-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.395057 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.394991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq77l\" (UniqueName: \"kubernetes.io/projected/f1491277-0af3-4314-910c-ccbf15867809-kube-api-access-zq77l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.395252 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.395118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1491277-0af3-4314-910c-ccbf15867809-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.395507 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.395489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1491277-0af3-4314-910c-ccbf15867809-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.403026 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.402993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq77l\" (UniqueName: \"kubernetes.io/projected/f1491277-0af3-4314-910c-ccbf15867809-kube-api-access-zq77l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.530069 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.529949 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:22.657333 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:22.657297 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh"] Jul 29 02:42:22.660598 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:42:22.660564 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1491277_0af3_4314_910c_ccbf15867809.slice/crio-8dccc28158e86fc45e845393726ea1a405c3d5b377630fdaec5e65892d3efdfb WatchSource:0}: Error finding container 8dccc28158e86fc45e845393726ea1a405c3d5b377630fdaec5e65892d3efdfb: Status 404 returned error can't find the container with id 8dccc28158e86fc45e845393726ea1a405c3d5b377630fdaec5e65892d3efdfb Jul 29 02:42:23.158451 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:23.158412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" event={"ID":"f1491277-0af3-4314-910c-ccbf15867809","Type":"ContainerStarted","Data":"8dccc28158e86fc45e845393726ea1a405c3d5b377630fdaec5e65892d3efdfb"} Jul 29 02:42:25.114336 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:25.114305 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-dc4c6" Jul 29 02:42:27.129232 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:27.129201 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:28.184085 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:28.184040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" event={"ID":"f1491277-0af3-4314-910c-ccbf15867809","Type":"ContainerStarted","Data":"04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094"} Jul 29 02:42:28.184532 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:28.184135 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:28.204883 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:28.204835 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" podStartSLOduration=1.4665873249999999 podStartE2EDuration="6.204819232s" totalImagesPullingTime="4.738231907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:42:28.202512766 +0000 UTC m=+438.262988151" watchObservedRunningTime="2026-07-29 02:42:28.204819232 +0000 UTC m=+438.265294616" Jul 29 02:42:31.150935 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:31.150902 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqpwv" Jul 29 02:42:39.191177 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:39.191097 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:40.863722 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.863680 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh"] Jul 29 02:42:40.864219 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.863880 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" podUID="f1491277-0af3-4314-910c-ccbf15867809" containerName="manager" containerID="cri-o://04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094" gracePeriod=2 Jul 29 02:42:40.871096 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.871069 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh"] Jul 29 02:42:40.882846 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.882819 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl"] Jul 29 02:42:40.883093 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.883066 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" containerName="manager" containerID="cri-o://1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c" gracePeriod=2 Jul 29 02:42:40.885919 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.885891 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p"] Jul 29 02:42:40.886407 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.886361 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1491277-0af3-4314-910c-ccbf15867809" containerName="manager" Jul 29 02:42:40.886495 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.886416 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1491277-0af3-4314-910c-ccbf15867809" containerName="manager" Jul 29 02:42:40.886533 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.886518 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1491277-0af3-4314-910c-ccbf15867809" containerName="manager" Jul 29 02:42:40.889625 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.889605 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:40.899531 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.899507 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl"] Jul 29 02:42:40.903393 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.903368 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p"] Jul 29 02:42:40.906206 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:40.906175 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5737718d-e914-4f3f-962f-fc38828a42f3\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-z4pnl\": pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:40.907851 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:40.907820 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:40.909282 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:40.909256 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:40.942180 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:40.942054 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:40.943180 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.943148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1772d863-042b-4e4e-bd78-16485dfb25c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2xh5p\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:40.943560 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:40.943542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgwf\" (UniqueName: \"kubernetes.io/projected/1772d863-042b-4e4e-bd78-16485dfb25c4-kube-api-access-trgwf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2xh5p\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:41.044388 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.044353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1772d863-042b-4e4e-bd78-16485dfb25c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2xh5p\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:41.044566 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.044407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgwf\" (UniqueName: \"kubernetes.io/projected/1772d863-042b-4e4e-bd78-16485dfb25c4-kube-api-access-trgwf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2xh5p\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:41.044757 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.044733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1772d863-042b-4e4e-bd78-16485dfb25c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2xh5p\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:41.053464 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.053409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgwf\" (UniqueName: \"kubernetes.io/projected/1772d863-042b-4e4e-bd78-16485dfb25c4-kube-api-access-trgwf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2xh5p\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:41.113192 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.113166 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.115041 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.114964 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:41.116334 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.116302 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.116584 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.116567 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:41.118394 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.118370 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.119796 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.119778 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:41.145180 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.145150 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq77l\" (UniqueName: \"kubernetes.io/projected/f1491277-0af3-4314-910c-ccbf15867809-kube-api-access-zq77l\") pod \"f1491277-0af3-4314-910c-ccbf15867809\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " Jul 29 02:42:41.145273 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.145186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1491277-0af3-4314-910c-ccbf15867809-extensions-socket-volume\") pod \"f1491277-0af3-4314-910c-ccbf15867809\" (UID: \"f1491277-0af3-4314-910c-ccbf15867809\") " Jul 29 02:42:41.145646 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.145618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1491277-0af3-4314-910c-ccbf15867809-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f1491277-0af3-4314-910c-ccbf15867809" (UID: "f1491277-0af3-4314-910c-ccbf15867809"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:42:41.147212 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.147187 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1491277-0af3-4314-910c-ccbf15867809-kube-api-access-zq77l" (OuterVolumeSpecName: "kube-api-access-zq77l") pod "f1491277-0af3-4314-910c-ccbf15867809" (UID: "f1491277-0af3-4314-910c-ccbf15867809"). InnerVolumeSpecName "kube-api-access-zq77l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:42:41.245676 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.245638 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zc6d\" (UniqueName: \"kubernetes.io/projected/5737718d-e914-4f3f-962f-fc38828a42f3-kube-api-access-2zc6d\") pod \"5737718d-e914-4f3f-962f-fc38828a42f3\" (UID: \"5737718d-e914-4f3f-962f-fc38828a42f3\") " Jul 29 02:42:41.245870 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.245820 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq77l\" (UniqueName: \"kubernetes.io/projected/f1491277-0af3-4314-910c-ccbf15867809-kube-api-access-zq77l\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:42:41.245870 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.245837 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1491277-0af3-4314-910c-ccbf15867809-extensions-socket-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:42:41.246366 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.246341 2579 generic.go:356] "Generic (PLEG): container finished" podID="5737718d-e914-4f3f-962f-fc38828a42f3" containerID="1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c" exitCode=0 Jul 29 02:42:41.246472 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.246406 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:41.246472 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.246430 2579 scope.go:117] "RemoveContainer" containerID="1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c" Jul 29 02:42:41.247874 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.247851 2579 generic.go:356] "Generic (PLEG): container finished" podID="f1491277-0af3-4314-910c-ccbf15867809" containerID="04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094" exitCode=0 Jul 29 02:42:41.247969 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.247903 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.248208 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.248170 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:41.248325 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.248249 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5737718d-e914-4f3f-962f-fc38828a42f3-kube-api-access-2zc6d" (OuterVolumeSpecName: "kube-api-access-2zc6d") pod "5737718d-e914-4f3f-962f-fc38828a42f3" (UID: "5737718d-e914-4f3f-962f-fc38828a42f3"). InnerVolumeSpecName "kube-api-access-2zc6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:42:41.249665 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.249641 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.251091 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.251063 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:41.252528 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.252500 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.255009 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.254995 2579 scope.go:117] "RemoveContainer" containerID="1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c" Jul 29 02:42:41.255294 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.255274 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c\": container with ID starting with 1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c not found: ID does not exist" containerID="1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c" Jul 29 02:42:41.255363 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.255303 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c"} err="failed to get container status \"1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c\": rpc error: code = NotFound desc = could not find container \"1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c\": container with ID starting with 1ca0b0cff77f4a196462ee6bfa032313b3d4c473e100ea27639ecc9e8598238c not found: ID does not exist" Jul 29 02:42:41.255363 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.255320 2579 scope.go:117] "RemoveContainer" containerID="04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094" Jul 29 02:42:41.259706 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.259683 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:41.262812 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.262785 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.263106 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.263091 2579 scope.go:117] "RemoveContainer" containerID="04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094" Jul 29 02:42:41.263403 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.263385 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094\": container with ID starting with 04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094 not found: ID does not exist" containerID="04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094" Jul 29 02:42:41.263454 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.263414 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094"} err="failed to get container status \"04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094\": rpc error: code = NotFound desc = could not find container \"04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094\": container with ID starting with 04ca9d8dcfe20b951fc0f8e38821aaad0ff3562e6401e91e5f7ec00ffc8dc094 not found: ID does not exist" Jul 29 02:42:41.282381 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.282350 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1491277_0af3_4314_910c_ccbf15867809.slice\": RecentStats: unable to find data in memory cache]" Jul 29 02:42:41.305475 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.305445 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:41.347056 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.347009 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zc6d\" (UniqueName: \"kubernetes.io/projected/5737718d-e914-4f3f-962f-fc38828a42f3-kube-api-access-2zc6d\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:42:41.435843 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:41.435816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p"] Jul 29 02:42:41.438043 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:42:41.438004 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1772d863_042b_4e4e_bd78_16485dfb25c4.slice/crio-13932928da0bc197afda9cca437d2d4c65258384b1e8351030daad7acc1c3e75 WatchSource:0}: Error finding container 13932928da0bc197afda9cca437d2d4c65258384b1e8351030daad7acc1c3e75: Status 404 returned error can't find the container with id 13932928da0bc197afda9cca437d2d4c65258384b1e8351030daad7acc1c3e75 Jul 29 02:42:41.556798 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.556764 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:41.558522 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:41.558501 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:42.259292 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:42.259233 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" event={"ID":"1772d863-042b-4e4e-bd78-16485dfb25c4","Type":"ContainerStarted","Data":"87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9"} Jul 29 02:42:42.259954 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:42.259929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" event={"ID":"1772d863-042b-4e4e-bd78-16485dfb25c4","Type":"ContainerStarted","Data":"13932928da0bc197afda9cca437d2d4c65258384b1e8351030daad7acc1c3e75"} Jul 29 02:42:42.260147 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:42.260116 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:42:42.281403 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:42.281356 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" podStartSLOduration=2.281343977 podStartE2EDuration="2.281343977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:42:42.279482731 +0000 UTC m=+452.339958110" watchObservedRunningTime="2026-07-29 02:42:42.281343977 +0000 UTC m=+452.341819433" Jul 29 02:42:42.281511 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:42.281374 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z4pnl\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z4pnl" Jul 29 02:42:42.283127 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:42:42.283104 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2d9nh\" is forbidden: User \"system:node:ip-10-0-141-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-222.ec2.internal' and this object" podUID="f1491277-0af3-4314-910c-ccbf15867809" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2d9nh" Jul 29 02:42:42.451861 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:42.451825 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" path="/var/lib/kubelet/pods/5737718d-e914-4f3f-962f-fc38828a42f3/volumes" Jul 29 02:42:42.452198 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:42.452184 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1491277-0af3-4314-910c-ccbf15867809" path="/var/lib/kubelet/pods/f1491277-0af3-4314-910c-ccbf15867809/volumes" Jul 29 02:42:54.272625 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:42:54.272594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:43:07.977198 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:07.977162 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p"] Jul 29 02:43:07.977620 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:07.977426 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" podUID="1772d863-042b-4e4e-bd78-16485dfb25c4" containerName="manager" containerID="cri-o://87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9" gracePeriod=10 Jul 29 02:43:08.184603 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.184571 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn"] Jul 29 02:43:08.184883 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.184871 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" containerName="manager" Jul 29 02:43:08.184883 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.184884 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" containerName="manager" Jul 29 02:43:08.184979 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.184945 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5737718d-e914-4f3f-962f-fc38828a42f3" containerName="manager" Jul 29 02:43:08.186905 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.186882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.204320 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.204297 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn"] Jul 29 02:43:08.209042 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.209008 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:43:08.286143 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.286047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/694141cf-7d91-47bb-a548-4b4f65555ce0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdggn\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.286143 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.286095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hpd\" (UniqueName: \"kubernetes.io/projected/694141cf-7d91-47bb-a548-4b4f65555ce0-kube-api-access-h6hpd\") pod \"kuadrant-operator-controller-manager-86878c769f-kdggn\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.375497 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.375464 2579 generic.go:356] "Generic (PLEG): container finished" podID="1772d863-042b-4e4e-bd78-16485dfb25c4" containerID="87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9" exitCode=0 Jul 29 02:43:08.375685 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.375505 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" Jul 29 02:43:08.375685 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.375550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" event={"ID":"1772d863-042b-4e4e-bd78-16485dfb25c4","Type":"ContainerDied","Data":"87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9"} Jul 29 02:43:08.375685 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.375608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p" event={"ID":"1772d863-042b-4e4e-bd78-16485dfb25c4","Type":"ContainerDied","Data":"13932928da0bc197afda9cca437d2d4c65258384b1e8351030daad7acc1c3e75"} Jul 29 02:43:08.375685 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.375637 2579 scope.go:117] "RemoveContainer" containerID="87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9" Jul 29 02:43:08.383897 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.383879 2579 scope.go:117] "RemoveContainer" containerID="87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9" Jul 29 02:43:08.384130 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:43:08.384113 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9\": container with ID starting with 87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9 not found: ID does not exist" containerID="87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9" Jul 29 02:43:08.384192 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.384144 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9"} err="failed to get container status \"87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9\": rpc error: code = NotFound desc = could not find container \"87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9\": container with ID starting with 87369b6fccdbbf56ab2dee9c8cd1e8ef321b62d85159b4db66e50b8160f692d9 not found: ID does not exist" Jul 29 02:43:08.386474 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.386457 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1772d863-042b-4e4e-bd78-16485dfb25c4-extensions-socket-volume\") pod \"1772d863-042b-4e4e-bd78-16485dfb25c4\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " Jul 29 02:43:08.386520 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.386498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trgwf\" (UniqueName: \"kubernetes.io/projected/1772d863-042b-4e4e-bd78-16485dfb25c4-kube-api-access-trgwf\") pod \"1772d863-042b-4e4e-bd78-16485dfb25c4\" (UID: \"1772d863-042b-4e4e-bd78-16485dfb25c4\") " Jul 29 02:43:08.386634 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.386621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/694141cf-7d91-47bb-a548-4b4f65555ce0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdggn\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.386667 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.386645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hpd\" (UniqueName: \"kubernetes.io/projected/694141cf-7d91-47bb-a548-4b4f65555ce0-kube-api-access-h6hpd\") pod \"kuadrant-operator-controller-manager-86878c769f-kdggn\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.386798 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.386777 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1772d863-042b-4e4e-bd78-16485dfb25c4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1772d863-042b-4e4e-bd78-16485dfb25c4" (UID: "1772d863-042b-4e4e-bd78-16485dfb25c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:43:08.386996 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.386980 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/694141cf-7d91-47bb-a548-4b4f65555ce0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdggn\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.388392 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.388372 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1772d863-042b-4e4e-bd78-16485dfb25c4-kube-api-access-trgwf" (OuterVolumeSpecName: "kube-api-access-trgwf") pod "1772d863-042b-4e4e-bd78-16485dfb25c4" (UID: "1772d863-042b-4e4e-bd78-16485dfb25c4"). InnerVolumeSpecName "kube-api-access-trgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:43:08.399163 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.399145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hpd\" (UniqueName: \"kubernetes.io/projected/694141cf-7d91-47bb-a548-4b4f65555ce0-kube-api-access-h6hpd\") pod \"kuadrant-operator-controller-manager-86878c769f-kdggn\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.487326 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.487294 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1772d863-042b-4e4e-bd78-16485dfb25c4-extensions-socket-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:43:08.487326 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.487320 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trgwf\" (UniqueName: \"kubernetes.io/projected/1772d863-042b-4e4e-bd78-16485dfb25c4-kube-api-access-trgwf\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:43:08.505354 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.505304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:08.636599 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.636513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn"] Jul 29 02:43:08.639610 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:43:08.639581 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694141cf_7d91_47bb_a548_4b4f65555ce0.slice/crio-052d93f7b4e7107cbaaaf6449116be0270355f769e07ae98eaf23a3e7bf7bee3 WatchSource:0}: Error finding container 052d93f7b4e7107cbaaaf6449116be0270355f769e07ae98eaf23a3e7bf7bee3: Status 404 returned error can't find the container with id 052d93f7b4e7107cbaaaf6449116be0270355f769e07ae98eaf23a3e7bf7bee3 Jul 29 02:43:08.692678 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.692654 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p"] Jul 29 02:43:08.695899 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:08.695876 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2xh5p"] Jul 29 02:43:09.383704 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:09.383668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" event={"ID":"694141cf-7d91-47bb-a548-4b4f65555ce0","Type":"ContainerStarted","Data":"aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba"} Jul 29 02:43:09.384129 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:09.383718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" event={"ID":"694141cf-7d91-47bb-a548-4b4f65555ce0","Type":"ContainerStarted","Data":"052d93f7b4e7107cbaaaf6449116be0270355f769e07ae98eaf23a3e7bf7bee3"} Jul 29 02:43:09.384129 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:09.383755 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:09.400574 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:09.400522 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" podStartSLOduration=1.400505323 podStartE2EDuration="1.400505323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:43:09.399502147 +0000 UTC m=+479.459977548" watchObservedRunningTime="2026-07-29 02:43:09.400505323 +0000 UTC m=+479.460980707" Jul 29 02:43:10.452588 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:10.452556 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1772d863-042b-4e4e-bd78-16485dfb25c4" path="/var/lib/kubelet/pods/1772d863-042b-4e4e-bd78-16485dfb25c4/volumes" Jul 29 02:43:20.390274 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:20.390239 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:43:24.340610 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.340580 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd"] Jul 29 02:43:24.340974 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.340960 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1772d863-042b-4e4e-bd78-16485dfb25c4" containerName="manager" Jul 29 02:43:24.341027 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.340975 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1772d863-042b-4e4e-bd78-16485dfb25c4" containerName="manager" Jul 29 02:43:24.341083 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.341071 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1772d863-042b-4e4e-bd78-16485dfb25c4" containerName="manager" Jul 29 02:43:24.351154 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.351131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.356662 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.356636 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rh4j7\"" Jul 29 02:43:24.363930 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.363907 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd"] Jul 29 02:43:24.416074 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416236 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416236 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrbl\" (UniqueName: \"kubernetes.io/projected/379ad44b-8b7d-4957-a529-1be09e159bbc-kube-api-access-ncrbl\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416236 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416236 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416236 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416236 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ad44b-8b7d-4957-a529-1be09e159bbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416444 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.416444 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.416346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517210 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517410 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517410 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517410 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517410 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrbl\" (UniqueName: \"kubernetes.io/projected/379ad44b-8b7d-4957-a529-1be09e159bbc-kube-api-access-ncrbl\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517624 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517624 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517624 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517624 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ad44b-8b7d-4957-a529-1be09e159bbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517851 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.517851 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.517778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.518122 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.518096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.518202 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.518158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.518272 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.518253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ad44b-8b7d-4957-a529-1be09e159bbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.519826 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.519799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.519942 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.519822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.527296 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.527276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379ad44b-8b7d-4957-a529-1be09e159bbc-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.527674 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.527659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrbl\" (UniqueName: \"kubernetes.io/projected/379ad44b-8b7d-4957-a529-1be09e159bbc-kube-api-access-ncrbl\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k5qpd\" (UID: \"379ad44b-8b7d-4957-a529-1be09e159bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.661355 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.661267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:24.799170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.799144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd"] Jul 29 02:43:24.801248 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:43:24.801221 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379ad44b_8b7d_4957_a529_1be09e159bbc.slice/crio-8a9fab834335b86f30dc67111c75a3cf982f6b687a65cdb66f6d4b947b8a2924 WatchSource:0}: Error finding container 8a9fab834335b86f30dc67111c75a3cf982f6b687a65cdb66f6d4b947b8a2924: Status 404 returned error can't find the container with id 8a9fab834335b86f30dc67111c75a3cf982f6b687a65cdb66f6d4b947b8a2924 Jul 29 02:43:24.803235 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.803208 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:43:24.803289 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.803270 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:43:24.803326 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:24.803298 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:43:25.458882 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:25.458846 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" event={"ID":"379ad44b-8b7d-4957-a529-1be09e159bbc","Type":"ContainerStarted","Data":"6b34b6a52296cbdc39997b8633ba6461938cc4cf0cd32a76daa59a3e26344ade"} Jul 29 02:43:25.459375 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:25.458898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" event={"ID":"379ad44b-8b7d-4957-a529-1be09e159bbc","Type":"ContainerStarted","Data":"8a9fab834335b86f30dc67111c75a3cf982f6b687a65cdb66f6d4b947b8a2924"} Jul 29 02:43:25.485094 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:25.485031 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" podStartSLOduration=1.485003441 podStartE2EDuration="1.485003441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:43:25.483541379 +0000 UTC m=+495.544016786" watchObservedRunningTime="2026-07-29 02:43:25.485003441 +0000 UTC m=+495.545478825" Jul 29 02:43:25.662055 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:25.662002 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:25.666732 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:25.666704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:26.463605 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:26.463578 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:26.464705 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:26.464686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:43:26.941818 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:26.941777 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-km579"] Jul 29 02:43:26.945181 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:26.945165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:26.947120 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:26.947096 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 02:43:26.947204 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:26.947119 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pj6th\"" Jul 29 02:43:26.952054 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:26.952031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-km579"] Jul 29 02:43:27.036625 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.036594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-km579"] Jul 29 02:43:27.040267 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.040238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwm6\" (UniqueName: \"kubernetes.io/projected/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-kube-api-access-dkwm6\") pod \"limitador-limitador-64c94fd64f-km579\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:27.040391 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.040299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-config-file\") pod \"limitador-limitador-64c94fd64f-km579\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:27.141437 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.141401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-config-file\") pod \"limitador-limitador-64c94fd64f-km579\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:27.141637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.141509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwm6\" (UniqueName: \"kubernetes.io/projected/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-kube-api-access-dkwm6\") pod \"limitador-limitador-64c94fd64f-km579\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:27.142878 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.142855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-config-file\") pod \"limitador-limitador-64c94fd64f-km579\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:27.149537 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.149506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwm6\" (UniqueName: \"kubernetes.io/projected/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-kube-api-access-dkwm6\") pod \"limitador-limitador-64c94fd64f-km579\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:27.255667 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.255580 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:27.383505 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.383476 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-km579"] Jul 29 02:43:27.385930 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:43:27.385904 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76748a1d_44f2_41c5_b4e2_07d456dfa9a9.slice/crio-8c20d64ab85152ca76503e38d53094d249003ab37460a0df4758307390a8526c WatchSource:0}: Error finding container 8c20d64ab85152ca76503e38d53094d249003ab37460a0df4758307390a8526c: Status 404 returned error can't find the container with id 8c20d64ab85152ca76503e38d53094d249003ab37460a0df4758307390a8526c Jul 29 02:43:27.473413 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.473371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" event={"ID":"76748a1d-44f2-41c5-b4e2-07d456dfa9a9","Type":"ContainerStarted","Data":"8c20d64ab85152ca76503e38d53094d249003ab37460a0df4758307390a8526c"} Jul 29 02:43:27.725475 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.725445 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-tnj84"] Jul 29 02:43:27.729927 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.729905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tnj84" Jul 29 02:43:27.731659 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.731635 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ntxjn\"" Jul 29 02:43:27.734430 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.734407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tnj84"] Jul 29 02:43:27.849463 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.849431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74m8\" (UniqueName: \"kubernetes.io/projected/40746ee3-9849-49e6-ad3c-a12213a0ee97-kube-api-access-s74m8\") pod \"authorino-58cd955675-tnj84\" (UID: \"40746ee3-9849-49e6-ad3c-a12213a0ee97\") " pod="kuadrant-system/authorino-58cd955675-tnj84" Jul 29 02:43:27.950796 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.950759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74m8\" (UniqueName: \"kubernetes.io/projected/40746ee3-9849-49e6-ad3c-a12213a0ee97-kube-api-access-s74m8\") pod \"authorino-58cd955675-tnj84\" (UID: \"40746ee3-9849-49e6-ad3c-a12213a0ee97\") " pod="kuadrant-system/authorino-58cd955675-tnj84" Jul 29 02:43:27.959089 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:27.959064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74m8\" (UniqueName: \"kubernetes.io/projected/40746ee3-9849-49e6-ad3c-a12213a0ee97-kube-api-access-s74m8\") pod \"authorino-58cd955675-tnj84\" (UID: \"40746ee3-9849-49e6-ad3c-a12213a0ee97\") " pod="kuadrant-system/authorino-58cd955675-tnj84" Jul 29 02:43:28.040620 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:28.040552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tnj84" Jul 29 02:43:28.195832 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:28.195780 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tnj84"] Jul 29 02:43:28.198859 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:43:28.198818 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40746ee3_9849_49e6_ad3c_a12213a0ee97.slice/crio-35001c0b8e92cd7119ce97271ad7117856c19e919603c2efdea4838984da0100 WatchSource:0}: Error finding container 35001c0b8e92cd7119ce97271ad7117856c19e919603c2efdea4838984da0100: Status 404 returned error can't find the container with id 35001c0b8e92cd7119ce97271ad7117856c19e919603c2efdea4838984da0100 Jul 29 02:43:28.480490 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:28.480448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tnj84" event={"ID":"40746ee3-9849-49e6-ad3c-a12213a0ee97","Type":"ContainerStarted","Data":"35001c0b8e92cd7119ce97271ad7117856c19e919603c2efdea4838984da0100"} Jul 29 02:43:32.504634 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:32.504589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" event={"ID":"76748a1d-44f2-41c5-b4e2-07d456dfa9a9","Type":"ContainerStarted","Data":"82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39"} Jul 29 02:43:32.505134 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:32.504744 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:32.506002 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:32.505981 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tnj84" event={"ID":"40746ee3-9849-49e6-ad3c-a12213a0ee97","Type":"ContainerStarted","Data":"e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999"} Jul 29 02:43:32.553166 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:32.553117 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" podStartSLOduration=2.044562589 podStartE2EDuration="6.553102124s" totalImagesPullingTime="4.508539535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:43:32.551464796 +0000 UTC m=+502.611940179" watchObservedRunningTime="2026-07-29 02:43:32.553102124 +0000 UTC m=+502.613577556" Jul 29 02:43:32.582150 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:32.582099 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-tnj84" podStartSLOduration=1.8801214000000002 podStartE2EDuration="5.582083292s" totalImagesPullingTime="3.701961892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:43:32.581763482 +0000 UTC m=+502.642238860" watchObservedRunningTime="2026-07-29 02:43:32.582083292 +0000 UTC m=+502.642558677" Jul 29 02:43:33.163702 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:33.163664 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tnj84"] Jul 29 02:43:34.514508 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:34.514452 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-tnj84" podUID="40746ee3-9849-49e6-ad3c-a12213a0ee97" containerName="authorino" containerID="cri-o://e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999" gracePeriod=30 Jul 29 02:43:34.761702 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:34.761679 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tnj84" Jul 29 02:43:34.814333 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:34.814253 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s74m8\" (UniqueName: \"kubernetes.io/projected/40746ee3-9849-49e6-ad3c-a12213a0ee97-kube-api-access-s74m8\") pod \"40746ee3-9849-49e6-ad3c-a12213a0ee97\" (UID: \"40746ee3-9849-49e6-ad3c-a12213a0ee97\") " Jul 29 02:43:34.816289 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:34.816260 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40746ee3-9849-49e6-ad3c-a12213a0ee97-kube-api-access-s74m8" (OuterVolumeSpecName: "kube-api-access-s74m8") pod "40746ee3-9849-49e6-ad3c-a12213a0ee97" (UID: "40746ee3-9849-49e6-ad3c-a12213a0ee97"). InnerVolumeSpecName "kube-api-access-s74m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:43:34.915106 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:34.915071 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74m8\" (UniqueName: \"kubernetes.io/projected/40746ee3-9849-49e6-ad3c-a12213a0ee97-kube-api-access-s74m8\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:43:35.520349 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.520316 2579 generic.go:356] "Generic (PLEG): container finished" podID="40746ee3-9849-49e6-ad3c-a12213a0ee97" containerID="e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999" exitCode=0 Jul 29 02:43:35.520349 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.520349 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tnj84" Jul 29 02:43:35.520809 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.520365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tnj84" event={"ID":"40746ee3-9849-49e6-ad3c-a12213a0ee97","Type":"ContainerDied","Data":"e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999"} Jul 29 02:43:35.520809 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.520394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tnj84" event={"ID":"40746ee3-9849-49e6-ad3c-a12213a0ee97","Type":"ContainerDied","Data":"35001c0b8e92cd7119ce97271ad7117856c19e919603c2efdea4838984da0100"} Jul 29 02:43:35.520809 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.520411 2579 scope.go:117] "RemoveContainer" containerID="e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999" Jul 29 02:43:35.528489 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.528472 2579 scope.go:117] "RemoveContainer" containerID="e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999" Jul 29 02:43:35.528745 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:43:35.528727 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999\": container with ID starting with e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999 not found: ID does not exist" containerID="e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999" Jul 29 02:43:35.528808 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.528752 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999"} err="failed to get container status \"e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999\": rpc error: code = NotFound desc = could not find container \"e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999\": container with ID starting with e07c31c9d8204fd838b6665519cb2d182f9622c3e33114563919b0914a76b999 not found: ID does not exist" Jul 29 02:43:35.546309 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.546283 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tnj84"] Jul 29 02:43:35.550065 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:35.550046 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-tnj84"] Jul 29 02:43:36.453603 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:36.453561 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40746ee3-9849-49e6-ad3c-a12213a0ee97" path="/var/lib/kubelet/pods/40746ee3-9849-49e6-ad3c-a12213a0ee97/volumes" Jul 29 02:43:43.511542 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:43.511506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:44.167035 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.166982 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-km579"] Jul 29 02:43:44.167216 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.167183 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" podUID="76748a1d-44f2-41c5-b4e2-07d456dfa9a9" containerName="limitador" containerID="cri-o://82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39" gracePeriod=30 Jul 29 02:43:44.714587 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.714559 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:44.804856 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.804765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-config-file\") pod \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " Jul 29 02:43:44.805052 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.804864 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkwm6\" (UniqueName: \"kubernetes.io/projected/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-kube-api-access-dkwm6\") pod \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\" (UID: \"76748a1d-44f2-41c5-b4e2-07d456dfa9a9\") " Jul 29 02:43:44.805141 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.805117 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-config-file" (OuterVolumeSpecName: "config-file") pod "76748a1d-44f2-41c5-b4e2-07d456dfa9a9" (UID: "76748a1d-44f2-41c5-b4e2-07d456dfa9a9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 02:43:44.806881 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.806859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-kube-api-access-dkwm6" (OuterVolumeSpecName: "kube-api-access-dkwm6") pod "76748a1d-44f2-41c5-b4e2-07d456dfa9a9" (UID: "76748a1d-44f2-41c5-b4e2-07d456dfa9a9"). InnerVolumeSpecName "kube-api-access-dkwm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:43:44.905413 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.905376 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkwm6\" (UniqueName: \"kubernetes.io/projected/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-kube-api-access-dkwm6\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:43:44.905413 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:44.905409 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/76748a1d-44f2-41c5-b4e2-07d456dfa9a9-config-file\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:43:45.568993 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.568953 2579 generic.go:356] "Generic (PLEG): container finished" podID="76748a1d-44f2-41c5-b4e2-07d456dfa9a9" containerID="82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39" exitCode=0 Jul 29 02:43:45.569178 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.569000 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" Jul 29 02:43:45.569178 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.569003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" event={"ID":"76748a1d-44f2-41c5-b4e2-07d456dfa9a9","Type":"ContainerDied","Data":"82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39"} Jul 29 02:43:45.569178 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.569081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-km579" event={"ID":"76748a1d-44f2-41c5-b4e2-07d456dfa9a9","Type":"ContainerDied","Data":"8c20d64ab85152ca76503e38d53094d249003ab37460a0df4758307390a8526c"} Jul 29 02:43:45.569178 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.569110 2579 scope.go:117] "RemoveContainer" containerID="82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39" Jul 29 02:43:45.577553 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.577535 2579 scope.go:117] "RemoveContainer" containerID="82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39" Jul 29 02:43:45.577837 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:43:45.577816 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39\": container with ID starting with 82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39 not found: ID does not exist" containerID="82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39" Jul 29 02:43:45.577881 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.577846 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39"} err="failed to get container status \"82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39\": rpc error: code = NotFound desc = could not find container \"82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39\": container with ID starting with 82427c19c8dd5504425195283bd5d9c4c58816fc52cbe9eda05c3b63bbc3ef39 not found: ID does not exist" Jul 29 02:43:45.588930 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.588899 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-km579"] Jul 29 02:43:45.592766 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:45.592743 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-km579"] Jul 29 02:43:46.452733 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:46.452699 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76748a1d-44f2-41c5-b4e2-07d456dfa9a9" path="/var/lib/kubelet/pods/76748a1d-44f2-41c5-b4e2-07d456dfa9a9/volumes" Jul 29 02:43:48.157887 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.157844 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2kncg"] Jul 29 02:43:48.158269 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.158256 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40746ee3-9849-49e6-ad3c-a12213a0ee97" containerName="authorino" Jul 29 02:43:48.158269 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.158270 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40746ee3-9849-49e6-ad3c-a12213a0ee97" containerName="authorino" Jul 29 02:43:48.158347 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.158281 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76748a1d-44f2-41c5-b4e2-07d456dfa9a9" containerName="limitador" Jul 29 02:43:48.158347 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.158287 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76748a1d-44f2-41c5-b4e2-07d456dfa9a9" containerName="limitador" Jul 29 02:43:48.158347 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.158338 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40746ee3-9849-49e6-ad3c-a12213a0ee97" containerName="authorino" Jul 29 02:43:48.158347 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.158347 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76748a1d-44f2-41c5-b4e2-07d456dfa9a9" containerName="limitador" Jul 29 02:43:48.162650 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.162627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.164637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.164612 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 02:43:48.164637 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.164633 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 29 02:43:48.165096 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.165077 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 02:43:48.165096 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.165092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bvt4v\"" Jul 29 02:43:48.168635 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.168615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2kncg"] Jul 29 02:43:48.337567 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.337529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21303fd6-e73c-42b4-bbbc-5f0a0cff26d2-data\") pod \"postgres-69fb4594cf-2kncg\" (UID: \"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.337761 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.337579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8s5\" (UniqueName: \"kubernetes.io/projected/21303fd6-e73c-42b4-bbbc-5f0a0cff26d2-kube-api-access-vl8s5\") pod \"postgres-69fb4594cf-2kncg\" (UID: \"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.438964 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.438930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8s5\" (UniqueName: \"kubernetes.io/projected/21303fd6-e73c-42b4-bbbc-5f0a0cff26d2-kube-api-access-vl8s5\") pod \"postgres-69fb4594cf-2kncg\" (UID: \"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.439144 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.439001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21303fd6-e73c-42b4-bbbc-5f0a0cff26d2-data\") pod \"postgres-69fb4594cf-2kncg\" (UID: \"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.439339 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.439319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/21303fd6-e73c-42b4-bbbc-5f0a0cff26d2-data\") pod \"postgres-69fb4594cf-2kncg\" (UID: \"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.446851 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.446829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8s5\" (UniqueName: \"kubernetes.io/projected/21303fd6-e73c-42b4-bbbc-5f0a0cff26d2-kube-api-access-vl8s5\") pod \"postgres-69fb4594cf-2kncg\" (UID: \"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.474472 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.474443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:48.604914 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:48.604889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2kncg"] Jul 29 02:43:48.607338 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:43:48.607314 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21303fd6_e73c_42b4_bbbc_5f0a0cff26d2.slice/crio-bef11a4481bdc97c0e0d74c278166c2ceaae6ca7bb928e102a9d5829856d6a4c WatchSource:0}: Error finding container bef11a4481bdc97c0e0d74c278166c2ceaae6ca7bb928e102a9d5829856d6a4c: Status 404 returned error can't find the container with id bef11a4481bdc97c0e0d74c278166c2ceaae6ca7bb928e102a9d5829856d6a4c Jul 29 02:43:49.591225 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:49.591172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" event={"ID":"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2","Type":"ContainerStarted","Data":"bef11a4481bdc97c0e0d74c278166c2ceaae6ca7bb928e102a9d5829856d6a4c"} Jul 29 02:43:53.611169 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:53.611080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" event={"ID":"21303fd6-e73c-42b4-bbbc-5f0a0cff26d2","Type":"ContainerStarted","Data":"61b13a8ad3ef1248399ce6a229f046cfcc87642fd70db16ca235c86351530304"} Jul 29 02:43:53.611511 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:53.611172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:43:53.627294 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:53.627247 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" podStartSLOduration=0.959761334 podStartE2EDuration="5.627233232s" totalImagesPullingTime="4.667471898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:43:53.625166778 +0000 UTC m=+523.685642163" watchObservedRunningTime="2026-07-29 02:43:53.627233232 +0000 UTC m=+523.687708617" Jul 29 02:43:59.643775 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:43:59.643747 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2kncg" Jul 29 02:44:00.200298 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.200261 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k6zb6"] Jul 29 02:44:00.203959 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.203936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" Jul 29 02:44:00.206200 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.206177 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ntxjn\"" Jul 29 02:44:00.214143 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.214123 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k6zb6"] Jul 29 02:44:00.232964 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.232938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47gg\" (UniqueName: \"kubernetes.io/projected/9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc-kube-api-access-k47gg\") pod \"authorino-5d4b67c446-k6zb6\" (UID: \"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc\") " pod="kuadrant-system/authorino-5d4b67c446-k6zb6" Jul 29 02:44:00.333521 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.333485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47gg\" (UniqueName: \"kubernetes.io/projected/9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc-kube-api-access-k47gg\") pod \"authorino-5d4b67c446-k6zb6\" (UID: \"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc\") " pod="kuadrant-system/authorino-5d4b67c446-k6zb6" Jul 29 02:44:00.342076 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.342040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47gg\" (UniqueName: \"kubernetes.io/projected/9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc-kube-api-access-k47gg\") pod \"authorino-5d4b67c446-k6zb6\" (UID: \"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc\") " pod="kuadrant-system/authorino-5d4b67c446-k6zb6" Jul 29 02:44:00.417300 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.417258 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k6zb6"] Jul 29 02:44:00.417533 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.417520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" Jul 29 02:44:00.442732 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.442701 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58986c6557-c6kpz"] Jul 29 02:44:00.449003 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.448977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58986c6557-c6kpz" Jul 29 02:44:00.457115 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.457052 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58986c6557-c6kpz"] Jul 29 02:44:00.535444 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.535409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rq2x\" (UniqueName: \"kubernetes.io/projected/bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd-kube-api-access-8rq2x\") pod \"authorino-58986c6557-c6kpz\" (UID: \"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd\") " pod="kuadrant-system/authorino-58986c6557-c6kpz" Jul 29 02:44:00.559331 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.559226 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k6zb6"] Jul 29 02:44:00.562953 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:44:00.562924 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b50306c_22f2_4f1d_9dd7_a5e0deea2fcc.slice/crio-21d0f351f923e4e4b86f45105ad9e509d4a56f31462265e401ff4ef6bda9e8bc WatchSource:0}: Error finding container 21d0f351f923e4e4b86f45105ad9e509d4a56f31462265e401ff4ef6bda9e8bc: Status 404 returned error can't find the container with id 21d0f351f923e4e4b86f45105ad9e509d4a56f31462265e401ff4ef6bda9e8bc Jul 29 02:44:00.636364 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.636330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rq2x\" (UniqueName: \"kubernetes.io/projected/bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd-kube-api-access-8rq2x\") pod \"authorino-58986c6557-c6kpz\" (UID: \"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd\") " pod="kuadrant-system/authorino-58986c6557-c6kpz" Jul 29 02:44:00.642880 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.642847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" event={"ID":"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc","Type":"ContainerStarted","Data":"21d0f351f923e4e4b86f45105ad9e509d4a56f31462265e401ff4ef6bda9e8bc"} Jul 29 02:44:00.646114 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.646091 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rq2x\" (UniqueName: \"kubernetes.io/projected/bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd-kube-api-access-8rq2x\") pod \"authorino-58986c6557-c6kpz\" (UID: \"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd\") " pod="kuadrant-system/authorino-58986c6557-c6kpz" Jul 29 02:44:00.762040 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.761943 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58986c6557-c6kpz" Jul 29 02:44:00.916245 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:00.916223 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58986c6557-c6kpz"] Jul 29 02:44:00.918430 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:44:00.918388 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdacf0bf_8cbd_4ac2_aef2_c3b564de54dd.slice/crio-fc9778c4c0fb6a819bb599dea5a56e47945e8fc25ae8e539ad44b88e066e611c WatchSource:0}: Error finding container fc9778c4c0fb6a819bb599dea5a56e47945e8fc25ae8e539ad44b88e066e611c: Status 404 returned error can't find the container with id fc9778c4c0fb6a819bb599dea5a56e47945e8fc25ae8e539ad44b88e066e611c Jul 29 02:44:01.081030 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.080989 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58986c6557-c6kpz"] Jul 29 02:44:01.118156 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.118119 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-688bbb4f77-ztxj2"] Jul 29 02:44:01.121599 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.121561 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.128114 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.128079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 02:44:01.136898 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.136873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-688bbb4f77-ztxj2"] Jul 29 02:44:01.140385 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.140358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9c35322-6a4c-4566-9971-bbf7b114c233-tls-cert\") pod \"authorino-688bbb4f77-ztxj2\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.140502 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.140398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bsg\" (UniqueName: \"kubernetes.io/projected/e9c35322-6a4c-4566-9971-bbf7b114c233-kube-api-access-m6bsg\") pod \"authorino-688bbb4f77-ztxj2\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.241439 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.241394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9c35322-6a4c-4566-9971-bbf7b114c233-tls-cert\") pod \"authorino-688bbb4f77-ztxj2\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.241640 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.241466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bsg\" (UniqueName: \"kubernetes.io/projected/e9c35322-6a4c-4566-9971-bbf7b114c233-kube-api-access-m6bsg\") pod \"authorino-688bbb4f77-ztxj2\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.243973 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.243950 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9c35322-6a4c-4566-9971-bbf7b114c233-tls-cert\") pod \"authorino-688bbb4f77-ztxj2\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.253432 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.253411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bsg\" (UniqueName: \"kubernetes.io/projected/e9c35322-6a4c-4566-9971-bbf7b114c233-kube-api-access-m6bsg\") pod \"authorino-688bbb4f77-ztxj2\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.432038 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.431987 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:44:01.572395 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.572323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-688bbb4f77-ztxj2"] Jul 29 02:44:01.575926 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:44:01.575896 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9c35322_6a4c_4566_9971_bbf7b114c233.slice/crio-ada6a6b27f24e1460f555f759a1309389d06bf4e1cf5791fa7e21bd5bd12b2bc WatchSource:0}: Error finding container ada6a6b27f24e1460f555f759a1309389d06bf4e1cf5791fa7e21bd5bd12b2bc: Status 404 returned error can't find the container with id ada6a6b27f24e1460f555f759a1309389d06bf4e1cf5791fa7e21bd5bd12b2bc Jul 29 02:44:01.648155 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.648119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" event={"ID":"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc","Type":"ContainerStarted","Data":"bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c"} Jul 29 02:44:01.648618 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.648169 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" podUID="9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc" containerName="authorino" containerID="cri-o://bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c" gracePeriod=30 Jul 29 02:44:01.649953 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.649925 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58986c6557-c6kpz" podUID="bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd" containerName="authorino" containerID="cri-o://636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5" gracePeriod=30 Jul 29 02:44:01.649953 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.649941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58986c6557-c6kpz" event={"ID":"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd","Type":"ContainerStarted","Data":"636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5"} Jul 29 02:44:01.650145 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.649975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58986c6557-c6kpz" event={"ID":"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd","Type":"ContainerStarted","Data":"fc9778c4c0fb6a819bb599dea5a56e47945e8fc25ae8e539ad44b88e066e611c"} Jul 29 02:44:01.651521 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.651485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" event={"ID":"e9c35322-6a4c-4566-9971-bbf7b114c233","Type":"ContainerStarted","Data":"ada6a6b27f24e1460f555f759a1309389d06bf4e1cf5791fa7e21bd5bd12b2bc"} Jul 29 02:44:01.663924 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.663879 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" podStartSLOduration=1.232168306 podStartE2EDuration="1.663862195s" totalImagesPullingTime="431.693889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:44:01.66255242 +0000 UTC m=+531.723027807" watchObservedRunningTime="2026-07-29 02:44:01.663862195 +0000 UTC m=+531.724337581" Jul 29 02:44:01.679325 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.679272 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58986c6557-c6kpz" podStartSLOduration=1.294380913 podStartE2EDuration="1.679256852s" totalImagesPullingTime="384.875939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:44:01.677859632 +0000 UTC m=+531.738335019" watchObservedRunningTime="2026-07-29 02:44:01.679256852 +0000 UTC m=+531.739732236" Jul 29 02:44:01.901971 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.901931 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58986c6557-c6kpz" Jul 29 02:44:01.910169 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.910149 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" Jul 29 02:44:01.946698 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.946668 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rq2x\" (UniqueName: \"kubernetes.io/projected/bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd-kube-api-access-8rq2x\") pod \"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd\" (UID: \"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd\") " Jul 29 02:44:01.946868 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.946712 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k47gg\" (UniqueName: \"kubernetes.io/projected/9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc-kube-api-access-k47gg\") pod \"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc\" (UID: \"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc\") " Jul 29 02:44:01.948714 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.948691 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc-kube-api-access-k47gg" (OuterVolumeSpecName: "kube-api-access-k47gg") pod "9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc" (UID: "9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc"). InnerVolumeSpecName "kube-api-access-k47gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:44:01.948824 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:01.948785 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd-kube-api-access-8rq2x" (OuterVolumeSpecName: "kube-api-access-8rq2x") pod "bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd" (UID: "bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd"). InnerVolumeSpecName "kube-api-access-8rq2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:44:02.047279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.047243 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rq2x\" (UniqueName: \"kubernetes.io/projected/bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd-kube-api-access-8rq2x\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:44:02.047279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.047274 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47gg\" (UniqueName: \"kubernetes.io/projected/9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc-kube-api-access-k47gg\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:44:02.657134 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.657039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" event={"ID":"e9c35322-6a4c-4566-9971-bbf7b114c233","Type":"ContainerStarted","Data":"f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce"} Jul 29 02:44:02.658319 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.658287 2579 generic.go:356] "Generic (PLEG): container finished" podID="9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc" containerID="bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c" exitCode=0 Jul 29 02:44:02.658453 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.658323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" event={"ID":"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc","Type":"ContainerDied","Data":"bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c"} Jul 29 02:44:02.658453 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.658366 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" event={"ID":"9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc","Type":"ContainerDied","Data":"21d0f351f923e4e4b86f45105ad9e509d4a56f31462265e401ff4ef6bda9e8bc"} Jul 29 02:44:02.658453 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.658319 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k6zb6" Jul 29 02:44:02.658453 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.658397 2579 scope.go:117] "RemoveContainer" containerID="bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c" Jul 29 02:44:02.659689 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.659669 2579 generic.go:356] "Generic (PLEG): container finished" podID="bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd" containerID="636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5" exitCode=0 Jul 29 02:44:02.659772 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.659708 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58986c6557-c6kpz" Jul 29 02:44:02.659772 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.659717 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58986c6557-c6kpz" event={"ID":"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd","Type":"ContainerDied","Data":"636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5"} Jul 29 02:44:02.659772 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.659744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58986c6557-c6kpz" event={"ID":"bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd","Type":"ContainerDied","Data":"fc9778c4c0fb6a819bb599dea5a56e47945e8fc25ae8e539ad44b88e066e611c"} Jul 29 02:44:02.666745 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.666727 2579 scope.go:117] "RemoveContainer" containerID="bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c" Jul 29 02:44:02.666996 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:44:02.666980 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c\": container with ID starting with bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c not found: ID does not exist" containerID="bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c" Jul 29 02:44:02.667077 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.667004 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c"} err="failed to get container status \"bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c\": rpc error: code = NotFound desc = could not find container \"bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c\": container with ID starting with bb0f904762a1aea0d5d782c8e6b0353ee967046011790cef1946fe0afc0fa07c not found: ID does not exist" Jul 29 02:44:02.667077 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.667039 2579 scope.go:117] "RemoveContainer" containerID="636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5" Jul 29 02:44:02.673930 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.673916 2579 scope.go:117] "RemoveContainer" containerID="636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5" Jul 29 02:44:02.674185 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:44:02.674169 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5\": container with ID starting with 636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5 not found: ID does not exist" containerID="636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5" Jul 29 02:44:02.674231 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.674190 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5"} err="failed to get container status \"636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5\": rpc error: code = NotFound desc = could not find container \"636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5\": container with ID starting with 636e5d74ca9ec230ddf2601e97f20d629bb60ddace3634768760e38c1d2efdb5 not found: ID does not exist" Jul 29 02:44:02.679285 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.679244 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" podStartSLOduration=1.303632511 podStartE2EDuration="1.679230518s" totalImagesPullingTime="375.598007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:44:02.677451454 +0000 UTC m=+532.737926849" watchObservedRunningTime="2026-07-29 02:44:02.679230518 +0000 UTC m=+532.739705902" Jul 29 02:44:02.699065 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.699040 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58986c6557-c6kpz"] Jul 29 02:44:02.701623 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.701602 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58986c6557-c6kpz"] Jul 29 02:44:02.712475 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.712453 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k6zb6"] Jul 29 02:44:02.726301 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:02.726277 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k6zb6"] Jul 29 02:44:04.452044 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:04.451963 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc" path="/var/lib/kubelet/pods/9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc/volumes" Jul 29 02:44:04.452376 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:04.452358 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd" path="/var/lib/kubelet/pods/bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd/volumes" Jul 29 02:44:07.484100 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.484061 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b54d569bf-bsfcj"] Jul 29 02:44:07.484469 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.484442 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc" containerName="authorino" Jul 29 02:44:07.484469 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.484455 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc" containerName="authorino" Jul 29 02:44:07.484469 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.484466 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd" containerName="authorino" Jul 29 02:44:07.484568 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.484471 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd" containerName="authorino" Jul 29 02:44:07.484568 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.484520 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b50306c-22f2-4f1d-9dd7-a5e0deea2fcc" containerName="authorino" Jul 29 02:44:07.484568 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.484530 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdacf0bf-8cbd-4ac2-aef2-c3b564de54dd" containerName="authorino" Jul 29 02:44:07.487665 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.487643 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.489618 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.489602 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pfhrm\"" Jul 29 02:44:07.490225 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.490205 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 02:44:07.490318 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.490301 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 02:44:07.495679 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.495653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-bsfcj"] Jul 29 02:44:07.592716 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.592674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdtpk\" (UniqueName: \"kubernetes.io/projected/5e0783f5-e132-4ecb-a222-f7db87bdc055-kube-api-access-jdtpk\") pod \"maas-controller-7b54d569bf-bsfcj\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.592864 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.592748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e0783f5-e132-4ecb-a222-f7db87bdc055-webhook-cert\") pod \"maas-controller-7b54d569bf-bsfcj\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.693366 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.693335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e0783f5-e132-4ecb-a222-f7db87bdc055-webhook-cert\") pod \"maas-controller-7b54d569bf-bsfcj\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.693501 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.693419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpk\" (UniqueName: \"kubernetes.io/projected/5e0783f5-e132-4ecb-a222-f7db87bdc055-kube-api-access-jdtpk\") pod \"maas-controller-7b54d569bf-bsfcj\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.695650 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.695630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e0783f5-e132-4ecb-a222-f7db87bdc055-webhook-cert\") pod \"maas-controller-7b54d569bf-bsfcj\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.700567 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.700544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdtpk\" (UniqueName: \"kubernetes.io/projected/5e0783f5-e132-4ecb-a222-f7db87bdc055-kube-api-access-jdtpk\") pod \"maas-controller-7b54d569bf-bsfcj\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.798823 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.798731 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:07.926652 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:07.926627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-bsfcj"] Jul 29 02:44:07.929225 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:44:07.929196 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0783f5_e132_4ecb_a222_f7db87bdc055.slice/crio-d0f731e49c2b44bd316cde2756fe70e52408545a32a65f6cbc9e7ff45882f7a2 WatchSource:0}: Error finding container d0f731e49c2b44bd316cde2756fe70e52408545a32a65f6cbc9e7ff45882f7a2: Status 404 returned error can't find the container with id d0f731e49c2b44bd316cde2756fe70e52408545a32a65f6cbc9e7ff45882f7a2 Jul 29 02:44:08.691381 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:08.691342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" event={"ID":"5e0783f5-e132-4ecb-a222-f7db87bdc055","Type":"ContainerStarted","Data":"d0f731e49c2b44bd316cde2756fe70e52408545a32a65f6cbc9e7ff45882f7a2"} Jul 29 02:44:10.701238 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:10.701205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" event={"ID":"5e0783f5-e132-4ecb-a222-f7db87bdc055","Type":"ContainerStarted","Data":"36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010"} Jul 29 02:44:10.701792 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:10.701349 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:10.721124 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:10.721076 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" podStartSLOduration=1.515597657 podStartE2EDuration="3.72105841s" totalImagesPullingTime="2.205460753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:44:10.720930679 +0000 UTC m=+540.781406063" watchObservedRunningTime="2026-07-29 02:44:10.72105841 +0000 UTC m=+540.781533796" Jul 29 02:44:21.707986 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:21.707953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:44:41.589193 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.589158 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh"] Jul 29 02:44:41.597650 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.597631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.601117 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.601092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8tntz\"" Jul 29 02:44:41.601555 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.601537 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 02:44:41.608342 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.608315 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh"] Jul 29 02:44:41.682983 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.682941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6czf\" (UniqueName: \"kubernetes.io/projected/53e82640-b1dd-4d27-acdb-7b365fef14fc-kube-api-access-n6czf\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.683170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.683042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e82640-b1dd-4d27-acdb-7b365fef14fc-maas-api-tls\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.683170 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.683075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e82640-b1dd-4d27-acdb-7b365fef14fc-openshift-service-ca\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.783484 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.783453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6czf\" (UniqueName: \"kubernetes.io/projected/53e82640-b1dd-4d27-acdb-7b365fef14fc-kube-api-access-n6czf\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.783655 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.783503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e82640-b1dd-4d27-acdb-7b365fef14fc-maas-api-tls\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.783655 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.783537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e82640-b1dd-4d27-acdb-7b365fef14fc-openshift-service-ca\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.784216 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.784192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e82640-b1dd-4d27-acdb-7b365fef14fc-openshift-service-ca\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.785963 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.785933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53e82640-b1dd-4d27-acdb-7b365fef14fc-maas-api-tls\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.790875 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.790850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6czf\" (UniqueName: \"kubernetes.io/projected/53e82640-b1dd-4d27-acdb-7b365fef14fc-kube-api-access-n6czf\") pod \"maas-api-58d5b6f47b-vg2zh\" (UID: \"53e82640-b1dd-4d27-acdb-7b365fef14fc\") " pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:41.909005 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:41.908928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:42.053946 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.053913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh"] Jul 29 02:44:42.056599 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:44:42.056570 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e82640_b1dd_4d27_acdb_7b365fef14fc.slice/crio-a9436f190f44a34c66c2af7bffd02b888289a1452a1222953dc8ce88aa981393 WatchSource:0}: Error finding container a9436f190f44a34c66c2af7bffd02b888289a1452a1222953dc8ce88aa981393: Status 404 returned error can't find the container with id a9436f190f44a34c66c2af7bffd02b888289a1452a1222953dc8ce88aa981393 Jul 29 02:44:42.161261 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.161182 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dnxkj"] Jul 29 02:44:42.166263 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.166244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.168227 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.168197 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 02:44:42.168339 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.168307 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mtmzq\"" Jul 29 02:44:42.174561 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.174539 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dnxkj"] Jul 29 02:44:42.288774 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.288737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d17922f-85e0-4da5-96b0-3b7f52be73e3-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dnxkj\" (UID: \"7d17922f-85e0-4da5-96b0-3b7f52be73e3\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.288946 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.288790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2vf\" (UniqueName: \"kubernetes.io/projected/7d17922f-85e0-4da5-96b0-3b7f52be73e3-kube-api-access-nz2vf\") pod \"payload-processing-5cc6cd444b-dnxkj\" (UID: \"7d17922f-85e0-4da5-96b0-3b7f52be73e3\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.390056 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.389994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d17922f-85e0-4da5-96b0-3b7f52be73e3-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dnxkj\" (UID: \"7d17922f-85e0-4da5-96b0-3b7f52be73e3\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.390242 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.390096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2vf\" (UniqueName: \"kubernetes.io/projected/7d17922f-85e0-4da5-96b0-3b7f52be73e3-kube-api-access-nz2vf\") pod \"payload-processing-5cc6cd444b-dnxkj\" (UID: \"7d17922f-85e0-4da5-96b0-3b7f52be73e3\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.390673 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.390652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d17922f-85e0-4da5-96b0-3b7f52be73e3-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dnxkj\" (UID: \"7d17922f-85e0-4da5-96b0-3b7f52be73e3\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.399604 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.399580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2vf\" (UniqueName: \"kubernetes.io/projected/7d17922f-85e0-4da5-96b0-3b7f52be73e3-kube-api-access-nz2vf\") pod \"payload-processing-5cc6cd444b-dnxkj\" (UID: \"7d17922f-85e0-4da5-96b0-3b7f52be73e3\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.478228 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.478179 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:42.606484 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.606388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dnxkj"] Jul 29 02:44:42.608557 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:44:42.608528 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d17922f_85e0_4da5_96b0_3b7f52be73e3.slice/crio-c5aa27ae590ebc28449353879b4b610a406c0129e5cb6b2f459f90a0c6eb5974 WatchSource:0}: Error finding container c5aa27ae590ebc28449353879b4b610a406c0129e5cb6b2f459f90a0c6eb5974: Status 404 returned error can't find the container with id c5aa27ae590ebc28449353879b4b610a406c0129e5cb6b2f459f90a0c6eb5974 Jul 29 02:44:42.845670 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.845584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" event={"ID":"53e82640-b1dd-4d27-acdb-7b365fef14fc","Type":"ContainerStarted","Data":"a9436f190f44a34c66c2af7bffd02b888289a1452a1222953dc8ce88aa981393"} Jul 29 02:44:42.847599 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:42.847569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" event={"ID":"7d17922f-85e0-4da5-96b0-3b7f52be73e3","Type":"ContainerStarted","Data":"c5aa27ae590ebc28449353879b4b610a406c0129e5cb6b2f459f90a0c6eb5974"} Jul 29 02:44:44.886261 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:44.886211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" event={"ID":"53e82640-b1dd-4d27-acdb-7b365fef14fc","Type":"ContainerStarted","Data":"dc341ca1c0c5dd466af7400a596daf0831d411370ccf0dd7cfc0be35a7f62f27"} Jul 29 02:44:44.886726 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:44.886302 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:44.903292 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:44.903235 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" podStartSLOduration=1.936057395 podStartE2EDuration="3.903220918s" totalImagesPullingTime="1.967163523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:44:44.901508087 +0000 UTC m=+574.961983472" watchObservedRunningTime="2026-07-29 02:44:44.903220918 +0000 UTC m=+574.963696315" Jul 29 02:44:45.893649 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:45.893542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" event={"ID":"7d17922f-85e0-4da5-96b0-3b7f52be73e3","Type":"ContainerStarted","Data":"f59f52186d7b72bad48f56203cd958438d91e1388120b8a34b97b54c38f47ce1"} Jul 29 02:44:45.894124 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:45.893876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:44:45.907631 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:45.907574 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" podStartSLOduration=1.022468138 podStartE2EDuration="3.90755553s" totalImagesPullingTime="2.885087392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:44:45.907263938 +0000 UTC m=+575.967739324" watchObservedRunningTime="2026-07-29 02:44:45.90755553 +0000 UTC m=+575.968030915" Jul 29 02:44:49.914349 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:49.914266 2579 generic.go:356] "Generic (PLEG): container finished" podID="379ad44b-8b7d-4957-a529-1be09e159bbc" containerID="6b34b6a52296cbdc39997b8633ba6461938cc4cf0cd32a76daa59a3e26344ade" exitCode=0 Jul 29 02:44:49.914349 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:49.914313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" event={"ID":"379ad44b-8b7d-4957-a529-1be09e159bbc","Type":"ContainerDied","Data":"6b34b6a52296cbdc39997b8633ba6461938cc4cf0cd32a76daa59a3e26344ade"} Jul 29 02:44:49.914770 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:49.914667 2579 scope.go:117] "RemoveContainer" containerID="6b34b6a52296cbdc39997b8633ba6461938cc4cf0cd32a76daa59a3e26344ade" Jul 29 02:44:49.915727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:49.915699 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:44:49.915783 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:49.915772 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:44:49.915820 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:49.915803 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 02:44:50.899046 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:50.898999 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d5b6f47b-vg2zh" Jul 29 02:44:50.921297 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:50.921269 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" event={"ID":"379ad44b-8b7d-4957-a529-1be09e159bbc","Type":"ContainerStarted","Data":"8fb9c898268573e33b1dc23643b13c67e150edba80c27bc376e3432d348a72ef"} Jul 29 02:44:51.662884 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:51.662845 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:44:51.663103 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:51.662978 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k5qpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 29 02:44:51.663103 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:51.663041 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" podUID="379ad44b-8b7d-4957-a529-1be09e159bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 29 02:44:52.662083 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:52.662038 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k5qpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 29 02:44:52.662622 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:52.662108 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" podUID="379ad44b-8b7d-4957-a529-1be09e159bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 29 02:44:53.661993 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:53.661954 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k5qpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 29 02:44:53.662290 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:53.662259 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" podUID="379ad44b-8b7d-4957-a529-1be09e159bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 29 02:44:54.666009 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:54.665978 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:44:54.666406 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:54.666253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:44:54.666967 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:54.666950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k5qpd" Jul 29 02:44:56.901375 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:44:56.901341 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-dnxkj" Jul 29 02:45:10.326912 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:10.326888 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:45:10.331267 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:10.331246 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:45:30.300659 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.300624 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5"] Jul 29 02:45:30.305617 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.305592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.308247 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.308223 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 29 02:45:30.308373 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.308226 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 02:45:30.308373 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.308333 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5lxvp\"" Jul 29 02:45:30.308491 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.308383 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 02:45:30.312955 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.312934 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5"] Jul 29 02:45:30.408141 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.408107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.408306 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.408150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.408306 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.408191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.408306 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.408210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncl7\" (UniqueName: \"kubernetes.io/projected/134849eb-a2f3-412a-88b8-4719949b0c2f-kube-api-access-vncl7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.408306 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.408295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/134849eb-a2f3-412a-88b8-4719949b0c2f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.408445 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.408388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509254 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509254 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509480 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509480 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509480 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncl7\" (UniqueName: \"kubernetes.io/projected/134849eb-a2f3-412a-88b8-4719949b0c2f-kube-api-access-vncl7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509480 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/134849eb-a2f3-412a-88b8-4719949b0c2f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509673 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509726 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.509726 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.509714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.511662 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.511634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/134849eb-a2f3-412a-88b8-4719949b0c2f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.511768 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.511749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/134849eb-a2f3-412a-88b8-4719949b0c2f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.516527 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.516506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncl7\" (UniqueName: \"kubernetes.io/projected/134849eb-a2f3-412a-88b8-4719949b0c2f-kube-api-access-vncl7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ttcf5\" (UID: \"134849eb-a2f3-412a-88b8-4719949b0c2f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.617974 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.617896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:30.743064 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.743041 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5"] Jul 29 02:45:30.747816 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:45:30.745476 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134849eb_a2f3_412a_88b8_4719949b0c2f.slice/crio-3f8797a78f939c5d38ff4c793892c723bc5689861f4fd671fc96acf74b0a19f9 WatchSource:0}: Error finding container 3f8797a78f939c5d38ff4c793892c723bc5689861f4fd671fc96acf74b0a19f9: Status 404 returned error can't find the container with id 3f8797a78f939c5d38ff4c793892c723bc5689861f4fd671fc96acf74b0a19f9 Jul 29 02:45:30.748860 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:30.748839 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 02:45:31.105320 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:31.105276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" event={"ID":"134849eb-a2f3-412a-88b8-4719949b0c2f","Type":"ContainerStarted","Data":"3f8797a78f939c5d38ff4c793892c723bc5689861f4fd671fc96acf74b0a19f9"} Jul 29 02:45:32.114702 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:32.113920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" event={"ID":"134849eb-a2f3-412a-88b8-4719949b0c2f","Type":"ContainerStarted","Data":"7599f442da9fa76707cc4b685846fb2cfbad1454384f653804d3929ef2a6a4ea"} Jul 29 02:45:32.134792 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:32.134737 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" podStartSLOduration=1.209828923 podStartE2EDuration="2.13472139s" totalImagesPullingTime="924.892467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:45:32.133713163 +0000 UTC m=+622.194188549" watchObservedRunningTime="2026-07-29 02:45:32.13472139 +0000 UTC m=+622.195196774" Jul 29 02:45:40.618907 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:40.618871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:40.619376 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:40.618928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:40.630890 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:40.630868 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:45:41.168497 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:45:41.168466 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ttcf5" Jul 29 02:46:55.444569 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.444537 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c5488cb64-smstx"] Jul 29 02:46:55.447924 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.447907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.454612 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.454588 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c5488cb64-smstx"] Jul 29 02:46:55.523350 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.523321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4g4\" (UniqueName: \"kubernetes.io/projected/499fd96b-aced-4e44-95e1-5b81f206a24e-kube-api-access-2c4g4\") pod \"authorino-5c5488cb64-smstx\" (UID: \"499fd96b-aced-4e44-95e1-5b81f206a24e\") " pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.523489 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.523358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/499fd96b-aced-4e44-95e1-5b81f206a24e-tls-cert\") pod \"authorino-5c5488cb64-smstx\" (UID: \"499fd96b-aced-4e44-95e1-5b81f206a24e\") " pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.624630 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.624595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4g4\" (UniqueName: \"kubernetes.io/projected/499fd96b-aced-4e44-95e1-5b81f206a24e-kube-api-access-2c4g4\") pod \"authorino-5c5488cb64-smstx\" (UID: \"499fd96b-aced-4e44-95e1-5b81f206a24e\") " pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.624782 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.624643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/499fd96b-aced-4e44-95e1-5b81f206a24e-tls-cert\") pod \"authorino-5c5488cb64-smstx\" (UID: \"499fd96b-aced-4e44-95e1-5b81f206a24e\") " pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.627040 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.627005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/499fd96b-aced-4e44-95e1-5b81f206a24e-tls-cert\") pod \"authorino-5c5488cb64-smstx\" (UID: \"499fd96b-aced-4e44-95e1-5b81f206a24e\") " pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.631053 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.631030 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4g4\" (UniqueName: \"kubernetes.io/projected/499fd96b-aced-4e44-95e1-5b81f206a24e-kube-api-access-2c4g4\") pod \"authorino-5c5488cb64-smstx\" (UID: \"499fd96b-aced-4e44-95e1-5b81f206a24e\") " pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.759068 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.758992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c5488cb64-smstx" Jul 29 02:46:55.888210 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:55.888182 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c5488cb64-smstx"] Jul 29 02:46:55.890540 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:46:55.890510 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod499fd96b_aced_4e44_95e1_5b81f206a24e.slice/crio-beeb95b7f1b38814d82a870e88c69e43c8a9457d4933bf747df33ab310afb723 WatchSource:0}: Error finding container beeb95b7f1b38814d82a870e88c69e43c8a9457d4933bf747df33ab310afb723: Status 404 returned error can't find the container with id beeb95b7f1b38814d82a870e88c69e43c8a9457d4933bf747df33ab310afb723 Jul 29 02:46:56.507480 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.507384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c5488cb64-smstx" event={"ID":"499fd96b-aced-4e44-95e1-5b81f206a24e","Type":"ContainerStarted","Data":"e7ead6169504260ec36e038d0d19bdbefca3ed4f093ed552e2be3dd3ccdc86d6"} Jul 29 02:46:56.507480 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.507451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c5488cb64-smstx" event={"ID":"499fd96b-aced-4e44-95e1-5b81f206a24e","Type":"ContainerStarted","Data":"beeb95b7f1b38814d82a870e88c69e43c8a9457d4933bf747df33ab310afb723"} Jul 29 02:46:56.524169 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.524123 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c5488cb64-smstx" podStartSLOduration=0.99566658 podStartE2EDuration="1.524108101s" totalImagesPullingTime="528.441521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:46:56.522419629 +0000 UTC m=+706.582895013" watchObservedRunningTime="2026-07-29 02:46:56.524108101 +0000 UTC m=+706.584583485" Jul 29 02:46:56.549838 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.549794 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-688bbb4f77-ztxj2"] Jul 29 02:46:56.550173 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.550144 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" podUID="e9c35322-6a4c-4566-9971-bbf7b114c233" containerName="authorino" containerID="cri-o://f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce" gracePeriod=30 Jul 29 02:46:56.796572 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.796549 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:46:56.838935 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.838904 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6bsg\" (UniqueName: \"kubernetes.io/projected/e9c35322-6a4c-4566-9971-bbf7b114c233-kube-api-access-m6bsg\") pod \"e9c35322-6a4c-4566-9971-bbf7b114c233\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " Jul 29 02:46:56.839113 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.838954 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9c35322-6a4c-4566-9971-bbf7b114c233-tls-cert\") pod \"e9c35322-6a4c-4566-9971-bbf7b114c233\" (UID: \"e9c35322-6a4c-4566-9971-bbf7b114c233\") " Jul 29 02:46:56.841057 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.841027 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c35322-6a4c-4566-9971-bbf7b114c233-kube-api-access-m6bsg" (OuterVolumeSpecName: "kube-api-access-m6bsg") pod "e9c35322-6a4c-4566-9971-bbf7b114c233" (UID: "e9c35322-6a4c-4566-9971-bbf7b114c233"). InnerVolumeSpecName "kube-api-access-m6bsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:46:56.852395 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.852367 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c35322-6a4c-4566-9971-bbf7b114c233-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e9c35322-6a4c-4566-9971-bbf7b114c233" (UID: "e9c35322-6a4c-4566-9971-bbf7b114c233"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:46:56.940284 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.940249 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9c35322-6a4c-4566-9971-bbf7b114c233-tls-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:46:56.940284 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:56.940277 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bsg\" (UniqueName: \"kubernetes.io/projected/e9c35322-6a4c-4566-9971-bbf7b114c233-kube-api-access-m6bsg\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:46:57.515044 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.514991 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9c35322-6a4c-4566-9971-bbf7b114c233" containerID="f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce" exitCode=0 Jul 29 02:46:57.515713 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.515691 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" Jul 29 02:46:57.517216 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.517180 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" event={"ID":"e9c35322-6a4c-4566-9971-bbf7b114c233","Type":"ContainerDied","Data":"f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce"} Jul 29 02:46:57.517361 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.517234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-688bbb4f77-ztxj2" event={"ID":"e9c35322-6a4c-4566-9971-bbf7b114c233","Type":"ContainerDied","Data":"ada6a6b27f24e1460f555f759a1309389d06bf4e1cf5791fa7e21bd5bd12b2bc"} Jul 29 02:46:57.517361 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.517258 2579 scope.go:117] "RemoveContainer" containerID="f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce" Jul 29 02:46:57.527478 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.527457 2579 scope.go:117] "RemoveContainer" containerID="f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce" Jul 29 02:46:57.527788 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:46:57.527769 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce\": container with ID starting with f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce not found: ID does not exist" containerID="f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce" Jul 29 02:46:57.527850 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.527800 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce"} err="failed to get container status \"f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce\": rpc error: code = NotFound desc = could not find container \"f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce\": container with ID starting with f5c5d84bdd93fd73ff7e0dd6a22ab44a13b01095436a4e8889460470d20a5bce not found: ID does not exist" Jul 29 02:46:57.535722 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.535697 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-688bbb4f77-ztxj2"] Jul 29 02:46:57.541652 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:57.541620 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-688bbb4f77-ztxj2"] Jul 29 02:46:58.452887 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:46:58.452855 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9c35322-6a4c-4566-9971-bbf7b114c233" path="/var/lib/kubelet/pods/e9c35322-6a4c-4566-9971-bbf7b114c233/volumes" Jul 29 02:47:43.031034 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.030991 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-bsfcj"] Jul 29 02:47:43.031860 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.031195 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" podUID="5e0783f5-e132-4ecb-a222-f7db87bdc055" containerName="manager" containerID="cri-o://36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010" gracePeriod=30 Jul 29 02:47:43.279502 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.279474 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:47:43.346763 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.346698 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdtpk\" (UniqueName: \"kubernetes.io/projected/5e0783f5-e132-4ecb-a222-f7db87bdc055-kube-api-access-jdtpk\") pod \"5e0783f5-e132-4ecb-a222-f7db87bdc055\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " Jul 29 02:47:43.346900 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.346765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e0783f5-e132-4ecb-a222-f7db87bdc055-webhook-cert\") pod \"5e0783f5-e132-4ecb-a222-f7db87bdc055\" (UID: \"5e0783f5-e132-4ecb-a222-f7db87bdc055\") " Jul 29 02:47:43.348786 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.348762 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0783f5-e132-4ecb-a222-f7db87bdc055-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5e0783f5-e132-4ecb-a222-f7db87bdc055" (UID: "5e0783f5-e132-4ecb-a222-f7db87bdc055"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:47:43.348876 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.348820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0783f5-e132-4ecb-a222-f7db87bdc055-kube-api-access-jdtpk" (OuterVolumeSpecName: "kube-api-access-jdtpk") pod "5e0783f5-e132-4ecb-a222-f7db87bdc055" (UID: "5e0783f5-e132-4ecb-a222-f7db87bdc055"). InnerVolumeSpecName "kube-api-access-jdtpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:47:43.447474 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.447441 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdtpk\" (UniqueName: \"kubernetes.io/projected/5e0783f5-e132-4ecb-a222-f7db87bdc055-kube-api-access-jdtpk\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:47:43.447474 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.447470 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e0783f5-e132-4ecb-a222-f7db87bdc055-webhook-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:47:43.728966 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.728933 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e0783f5-e132-4ecb-a222-f7db87bdc055" containerID="36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010" exitCode=0 Jul 29 02:47:43.729167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.728982 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" Jul 29 02:47:43.729167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.729028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" event={"ID":"5e0783f5-e132-4ecb-a222-f7db87bdc055","Type":"ContainerDied","Data":"36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010"} Jul 29 02:47:43.729167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.729086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-bsfcj" event={"ID":"5e0783f5-e132-4ecb-a222-f7db87bdc055","Type":"ContainerDied","Data":"d0f731e49c2b44bd316cde2756fe70e52408545a32a65f6cbc9e7ff45882f7a2"} Jul 29 02:47:43.729167 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.729108 2579 scope.go:117] "RemoveContainer" containerID="36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010" Jul 29 02:47:43.737526 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.737508 2579 scope.go:117] "RemoveContainer" containerID="36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010" Jul 29 02:47:43.737793 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:47:43.737775 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010\": container with ID starting with 36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010 not found: ID does not exist" containerID="36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010" Jul 29 02:47:43.737848 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.737808 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010"} err="failed to get container status \"36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010\": rpc error: code = NotFound desc = could not find container \"36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010\": container with ID starting with 36b9fff28584a11850beace1016da8f3327e47973096cec2355fa932ab04b010 not found: ID does not exist" Jul 29 02:47:43.748832 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.748808 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-bsfcj"] Jul 29 02:47:43.752114 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:43.752088 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-bsfcj"] Jul 29 02:47:44.451959 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:47:44.451918 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e0783f5-e132-4ecb-a222-f7db87bdc055" path="/var/lib/kubelet/pods/5e0783f5-e132-4ecb-a222-f7db87bdc055/volumes" Jul 29 02:48:44.523951 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.523912 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b54d569bf-td5sx"] Jul 29 02:48:44.524545 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.524424 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0783f5-e132-4ecb-a222-f7db87bdc055" containerName="manager" Jul 29 02:48:44.524545 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.524444 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0783f5-e132-4ecb-a222-f7db87bdc055" containerName="manager" Jul 29 02:48:44.524545 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.524470 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c35322-6a4c-4566-9971-bbf7b114c233" containerName="authorino" Jul 29 02:48:44.524545 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.524481 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c35322-6a4c-4566-9971-bbf7b114c233" containerName="authorino" Jul 29 02:48:44.524740 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.524571 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e0783f5-e132-4ecb-a222-f7db87bdc055" containerName="manager" Jul 29 02:48:44.524740 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.524585 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9c35322-6a4c-4566-9971-bbf7b114c233" containerName="authorino" Jul 29 02:48:44.527540 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.527519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.531299 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.531278 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 02:48:44.531376 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.531278 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pfhrm\"" Jul 29 02:48:44.531623 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.531610 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 02:48:44.537325 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.537306 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-td5sx"] Jul 29 02:48:44.558972 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.558934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2412597a-7636-4f4f-86e0-6bc7cc271360-webhook-cert\") pod \"maas-controller-7b54d569bf-td5sx\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.559088 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.559065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggdc\" (UniqueName: \"kubernetes.io/projected/2412597a-7636-4f4f-86e0-6bc7cc271360-kube-api-access-rggdc\") pod \"maas-controller-7b54d569bf-td5sx\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.659638 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.659605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggdc\" (UniqueName: \"kubernetes.io/projected/2412597a-7636-4f4f-86e0-6bc7cc271360-kube-api-access-rggdc\") pod \"maas-controller-7b54d569bf-td5sx\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.659788 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.659654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2412597a-7636-4f4f-86e0-6bc7cc271360-webhook-cert\") pod \"maas-controller-7b54d569bf-td5sx\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.662074 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.662047 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2412597a-7636-4f4f-86e0-6bc7cc271360-webhook-cert\") pod \"maas-controller-7b54d569bf-td5sx\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.667391 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.667367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggdc\" (UniqueName: \"kubernetes.io/projected/2412597a-7636-4f4f-86e0-6bc7cc271360-kube-api-access-rggdc\") pod \"maas-controller-7b54d569bf-td5sx\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.837466 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.837380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:44.962155 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:44.962128 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-td5sx"] Jul 29 02:48:44.963907 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:48:44.963884 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2412597a_7636_4f4f_86e0_6bc7cc271360.slice/crio-18603c4f6d7d456abdd7cc53fdd5cb8af03349060951a881acb096ce1bd15660 WatchSource:0}: Error finding container 18603c4f6d7d456abdd7cc53fdd5cb8af03349060951a881acb096ce1bd15660: Status 404 returned error can't find the container with id 18603c4f6d7d456abdd7cc53fdd5cb8af03349060951a881acb096ce1bd15660 Jul 29 02:48:45.013749 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:45.013715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-td5sx" event={"ID":"2412597a-7636-4f4f-86e0-6bc7cc271360","Type":"ContainerStarted","Data":"18603c4f6d7d456abdd7cc53fdd5cb8af03349060951a881acb096ce1bd15660"} Jul 29 02:48:46.019516 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:46.019480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-td5sx" event={"ID":"2412597a-7636-4f4f-86e0-6bc7cc271360","Type":"ContainerStarted","Data":"ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3"} Jul 29 02:48:46.019901 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:46.019598 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:48:46.035718 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:46.035671 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b54d569bf-td5sx" podStartSLOduration=1.577581322 podStartE2EDuration="2.035659211s" totalImagesPullingTime="458.077889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:48:46.033779819 +0000 UTC m=+816.094255204" watchObservedRunningTime="2026-07-29 02:48:46.035659211 +0000 UTC m=+816.096134614" Jul 29 02:48:57.025871 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:48:57.025838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:49:01.714764 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:01.714726 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-td5sx"] Jul 29 02:49:01.715159 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:01.714942 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b54d569bf-td5sx" podUID="2412597a-7636-4f4f-86e0-6bc7cc271360" containerName="manager" containerID="cri-o://ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3" gracePeriod=30 Jul 29 02:49:01.960174 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:01.960148 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:49:02.094513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.094441 2579 generic.go:356] "Generic (PLEG): container finished" podID="2412597a-7636-4f4f-86e0-6bc7cc271360" containerID="ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3" exitCode=0 Jul 29 02:49:02.094513 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.094482 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-td5sx" Jul 29 02:49:02.094677 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.094526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-td5sx" event={"ID":"2412597a-7636-4f4f-86e0-6bc7cc271360","Type":"ContainerDied","Data":"ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3"} Jul 29 02:49:02.094677 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.094574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-td5sx" event={"ID":"2412597a-7636-4f4f-86e0-6bc7cc271360","Type":"ContainerDied","Data":"18603c4f6d7d456abdd7cc53fdd5cb8af03349060951a881acb096ce1bd15660"} Jul 29 02:49:02.094677 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.094596 2579 scope.go:117] "RemoveContainer" containerID="ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3" Jul 29 02:49:02.095727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.095699 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2412597a-7636-4f4f-86e0-6bc7cc271360-webhook-cert\") pod \"2412597a-7636-4f4f-86e0-6bc7cc271360\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " Jul 29 02:49:02.095788 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.095740 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rggdc\" (UniqueName: \"kubernetes.io/projected/2412597a-7636-4f4f-86e0-6bc7cc271360-kube-api-access-rggdc\") pod \"2412597a-7636-4f4f-86e0-6bc7cc271360\" (UID: \"2412597a-7636-4f4f-86e0-6bc7cc271360\") " Jul 29 02:49:02.097676 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.097637 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2412597a-7636-4f4f-86e0-6bc7cc271360-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2412597a-7636-4f4f-86e0-6bc7cc271360" (UID: "2412597a-7636-4f4f-86e0-6bc7cc271360"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:49:02.097785 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.097702 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2412597a-7636-4f4f-86e0-6bc7cc271360-kube-api-access-rggdc" (OuterVolumeSpecName: "kube-api-access-rggdc") pod "2412597a-7636-4f4f-86e0-6bc7cc271360" (UID: "2412597a-7636-4f4f-86e0-6bc7cc271360"). InnerVolumeSpecName "kube-api-access-rggdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:49:02.112641 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.112622 2579 scope.go:117] "RemoveContainer" containerID="ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3" Jul 29 02:49:02.112911 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:49:02.112894 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3\": container with ID starting with ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3 not found: ID does not exist" containerID="ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3" Jul 29 02:49:02.112960 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.112918 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3"} err="failed to get container status \"ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3\": rpc error: code = NotFound desc = could not find container \"ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3\": container with ID starting with ad93ea0a3435c01f63ac6f72ff39bccf12b3475b17ebc2ad0376f9a9ac0552f3 not found: ID does not exist" Jul 29 02:49:02.196345 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.196324 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2412597a-7636-4f4f-86e0-6bc7cc271360-webhook-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:49:02.196345 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.196344 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rggdc\" (UniqueName: \"kubernetes.io/projected/2412597a-7636-4f4f-86e0-6bc7cc271360-kube-api-access-rggdc\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:49:02.414283 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.414259 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-td5sx"] Jul 29 02:49:02.417199 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.417180 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-td5sx"] Jul 29 02:49:02.452718 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:02.452687 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2412597a-7636-4f4f-86e0-6bc7cc271360" path="/var/lib/kubelet/pods/2412597a-7636-4f4f-86e0-6bc7cc271360/volumes" Jul 29 02:49:03.149880 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.149847 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b54d569bf-pwmlg"] Jul 29 02:49:03.150291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.150236 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2412597a-7636-4f4f-86e0-6bc7cc271360" containerName="manager" Jul 29 02:49:03.150291 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.150249 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2412597a-7636-4f4f-86e0-6bc7cc271360" containerName="manager" Jul 29 02:49:03.150373 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.150298 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2412597a-7636-4f4f-86e0-6bc7cc271360" containerName="manager" Jul 29 02:49:03.154970 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.154951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.157199 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.157169 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 02:49:03.157344 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.157256 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pfhrm\"" Jul 29 02:49:03.157344 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.157186 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 02:49:03.160231 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.160212 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-pwmlg"] Jul 29 02:49:03.304853 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.304818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glmgd\" (UniqueName: \"kubernetes.io/projected/81407d25-a23e-4c4f-8b86-59cc4bf3c539-kube-api-access-glmgd\") pod \"maas-controller-7b54d569bf-pwmlg\" (UID: \"81407d25-a23e-4c4f-8b86-59cc4bf3c539\") " pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.305000 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.304859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81407d25-a23e-4c4f-8b86-59cc4bf3c539-webhook-cert\") pod \"maas-controller-7b54d569bf-pwmlg\" (UID: \"81407d25-a23e-4c4f-8b86-59cc4bf3c539\") " pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.406142 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.406064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glmgd\" (UniqueName: \"kubernetes.io/projected/81407d25-a23e-4c4f-8b86-59cc4bf3c539-kube-api-access-glmgd\") pod \"maas-controller-7b54d569bf-pwmlg\" (UID: \"81407d25-a23e-4c4f-8b86-59cc4bf3c539\") " pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.406142 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.406103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81407d25-a23e-4c4f-8b86-59cc4bf3c539-webhook-cert\") pod \"maas-controller-7b54d569bf-pwmlg\" (UID: \"81407d25-a23e-4c4f-8b86-59cc4bf3c539\") " pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.408435 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.408413 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81407d25-a23e-4c4f-8b86-59cc4bf3c539-webhook-cert\") pod \"maas-controller-7b54d569bf-pwmlg\" (UID: \"81407d25-a23e-4c4f-8b86-59cc4bf3c539\") " pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.412741 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.412717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glmgd\" (UniqueName: \"kubernetes.io/projected/81407d25-a23e-4c4f-8b86-59cc4bf3c539-kube-api-access-glmgd\") pod \"maas-controller-7b54d569bf-pwmlg\" (UID: \"81407d25-a23e-4c4f-8b86-59cc4bf3c539\") " pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.467271 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.467244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:03.596296 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:03.596267 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b54d569bf-pwmlg"] Jul 29 02:49:03.599641 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:49:03.599616 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81407d25_a23e_4c4f_8b86_59cc4bf3c539.slice/crio-28ec0ace7647ab45e8a25803e1f49def8091a9a7b819532bc1fcfc841cbcad7d WatchSource:0}: Error finding container 28ec0ace7647ab45e8a25803e1f49def8091a9a7b819532bc1fcfc841cbcad7d: Status 404 returned error can't find the container with id 28ec0ace7647ab45e8a25803e1f49def8091a9a7b819532bc1fcfc841cbcad7d Jul 29 02:49:04.106652 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:04.106606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-pwmlg" event={"ID":"81407d25-a23e-4c4f-8b86-59cc4bf3c539","Type":"ContainerStarted","Data":"953c7e86adfce592283df4a4c72a63999d270c5d79669711d2dacf12c0fd0bf5"} Jul 29 02:49:04.106915 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:04.106674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b54d569bf-pwmlg" event={"ID":"81407d25-a23e-4c4f-8b86-59cc4bf3c539","Type":"ContainerStarted","Data":"28ec0ace7647ab45e8a25803e1f49def8091a9a7b819532bc1fcfc841cbcad7d"} Jul 29 02:49:04.106915 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:04.106720 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:49:04.121986 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:04.121897 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b54d569bf-pwmlg" podStartSLOduration=0.851541439 podStartE2EDuration="1.121874699s" totalImagesPullingTime="270.33326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:49:04.119282698 +0000 UTC m=+834.179758082" watchObservedRunningTime="2026-07-29 02:49:04.121874699 +0000 UTC m=+834.182350085" Jul 29 02:49:15.113056 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:49:15.113006 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b54d569bf-pwmlg" Jul 29 02:50:10.356840 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:50:10.356771 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:50:10.359989 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:50:10.359964 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:54:16.602675 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:16.602644 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn"] Jul 29 02:54:16.603388 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:16.602831 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" podUID="694141cf-7d91-47bb-a548-4b4f65555ce0" containerName="manager" containerID="cri-o://aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba" gracePeriod=10 Jul 29 02:54:18.054635 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.054611 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:54:18.186716 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.186693 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/694141cf-7d91-47bb-a548-4b4f65555ce0-extensions-socket-volume\") pod \"694141cf-7d91-47bb-a548-4b4f65555ce0\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " Jul 29 02:54:18.186847 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.186731 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6hpd\" (UniqueName: \"kubernetes.io/projected/694141cf-7d91-47bb-a548-4b4f65555ce0-kube-api-access-h6hpd\") pod \"694141cf-7d91-47bb-a548-4b4f65555ce0\" (UID: \"694141cf-7d91-47bb-a548-4b4f65555ce0\") " Jul 29 02:54:18.187050 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.187008 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694141cf-7d91-47bb-a548-4b4f65555ce0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "694141cf-7d91-47bb-a548-4b4f65555ce0" (UID: "694141cf-7d91-47bb-a548-4b4f65555ce0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:54:18.188705 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.188684 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694141cf-7d91-47bb-a548-4b4f65555ce0-kube-api-access-h6hpd" (OuterVolumeSpecName: "kube-api-access-h6hpd") pod "694141cf-7d91-47bb-a548-4b4f65555ce0" (UID: "694141cf-7d91-47bb-a548-4b4f65555ce0"). InnerVolumeSpecName "kube-api-access-h6hpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:54:18.287154 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.287132 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/694141cf-7d91-47bb-a548-4b4f65555ce0-extensions-socket-volume\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:54:18.287154 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.287151 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6hpd\" (UniqueName: \"kubernetes.io/projected/694141cf-7d91-47bb-a548-4b4f65555ce0-kube-api-access-h6hpd\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:54:18.544391 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.544359 2579 generic.go:356] "Generic (PLEG): container finished" podID="694141cf-7d91-47bb-a548-4b4f65555ce0" containerID="aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba" exitCode=0 Jul 29 02:54:18.544470 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.544421 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" Jul 29 02:54:18.544509 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.544455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" event={"ID":"694141cf-7d91-47bb-a548-4b4f65555ce0","Type":"ContainerDied","Data":"aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba"} Jul 29 02:54:18.544599 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.544584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn" event={"ID":"694141cf-7d91-47bb-a548-4b4f65555ce0","Type":"ContainerDied","Data":"052d93f7b4e7107cbaaaf6449116be0270355f769e07ae98eaf23a3e7bf7bee3"} Jul 29 02:54:18.544645 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.544628 2579 scope.go:117] "RemoveContainer" containerID="aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba" Jul 29 02:54:18.553401 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.553387 2579 scope.go:117] "RemoveContainer" containerID="aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba" Jul 29 02:54:18.553625 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:54:18.553607 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba\": container with ID starting with aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba not found: ID does not exist" containerID="aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba" Jul 29 02:54:18.553684 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.553630 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba"} err="failed to get container status \"aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba\": rpc error: code = NotFound desc = could not find container \"aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba\": container with ID starting with aeb46570e6215a33d2325e2e85ac3202ab6025740efb72bf3052ee670ab3c8ba not found: ID does not exist" Jul 29 02:54:18.559153 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.559133 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn"] Jul 29 02:54:18.562727 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:18.562709 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdggn"] Jul 29 02:54:20.452337 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:54:20.452307 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="694141cf-7d91-47bb-a548-4b4f65555ce0" path="/var/lib/kubelet/pods/694141cf-7d91-47bb-a548-4b4f65555ce0/volumes" Jul 29 02:55:10.386451 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:10.386422 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:55:10.388922 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:10.386550 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 02:55:17.767693 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.767658 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr"] Jul 29 02:55:17.768199 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.768180 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="694141cf-7d91-47bb-a548-4b4f65555ce0" containerName="manager" Jul 29 02:55:17.768275 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.768202 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="694141cf-7d91-47bb-a548-4b4f65555ce0" containerName="manager" Jul 29 02:55:17.768325 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.768300 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="694141cf-7d91-47bb-a548-4b4f65555ce0" containerName="manager" Jul 29 02:55:17.772315 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.772293 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:17.774118 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.774098 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nq5jt\"" Jul 29 02:55:17.780712 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.780688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr"] Jul 29 02:55:17.898076 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.898052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvlrq\" (UniqueName: \"kubernetes.io/projected/63a5d434-45db-443a-9c54-97a01baab316-kube-api-access-mvlrq\") pod \"kuadrant-operator-controller-manager-86878c769f-pzkxr\" (UID: \"63a5d434-45db-443a-9c54-97a01baab316\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:17.898163 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.898099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63a5d434-45db-443a-9c54-97a01baab316-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pzkxr\" (UID: \"63a5d434-45db-443a-9c54-97a01baab316\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:17.999097 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.999068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvlrq\" (UniqueName: \"kubernetes.io/projected/63a5d434-45db-443a-9c54-97a01baab316-kube-api-access-mvlrq\") pod \"kuadrant-operator-controller-manager-86878c769f-pzkxr\" (UID: \"63a5d434-45db-443a-9c54-97a01baab316\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:17.999203 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.999117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63a5d434-45db-443a-9c54-97a01baab316-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pzkxr\" (UID: \"63a5d434-45db-443a-9c54-97a01baab316\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:17.999395 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:17.999381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63a5d434-45db-443a-9c54-97a01baab316-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pzkxr\" (UID: \"63a5d434-45db-443a-9c54-97a01baab316\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:18.007214 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:18.007193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvlrq\" (UniqueName: \"kubernetes.io/projected/63a5d434-45db-443a-9c54-97a01baab316-kube-api-access-mvlrq\") pod \"kuadrant-operator-controller-manager-86878c769f-pzkxr\" (UID: \"63a5d434-45db-443a-9c54-97a01baab316\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:18.082517 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:18.082468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:18.221510 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:18.221472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr"] Jul 29 02:55:18.227946 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:55:18.227921 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a5d434_45db_443a_9c54_97a01baab316.slice/crio-964f54f8d0b22ffdd96eb3d1dbb305a4b93127ed32949ea833cd905ae5cca4cf WatchSource:0}: Error finding container 964f54f8d0b22ffdd96eb3d1dbb305a4b93127ed32949ea833cd905ae5cca4cf: Status 404 returned error can't find the container with id 964f54f8d0b22ffdd96eb3d1dbb305a4b93127ed32949ea833cd905ae5cca4cf Jul 29 02:55:18.793737 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:18.793702 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" event={"ID":"63a5d434-45db-443a-9c54-97a01baab316","Type":"ContainerStarted","Data":"fd6836cec8b4138fa6c6dd3167f5b49d3fc8a0ce45ddea3d2fc4ae525857e342"} Jul 29 02:55:18.794107 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:18.793760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" event={"ID":"63a5d434-45db-443a-9c54-97a01baab316","Type":"ContainerStarted","Data":"964f54f8d0b22ffdd96eb3d1dbb305a4b93127ed32949ea833cd905ae5cca4cf"} Jul 29 02:55:18.794107 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:18.793847 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:55:18.810628 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:18.810594 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" podStartSLOduration=1.810581019 podStartE2EDuration="1.810581019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:55:18.809160575 +0000 UTC m=+1208.869635959" watchObservedRunningTime="2026-07-29 02:55:18.810581019 +0000 UTC m=+1208.871056403" Jul 29 02:55:29.799694 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:55:29.799666 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pzkxr" Jul 29 02:58:08.393172 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.393141 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn"] Jul 29 02:58:08.396649 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.396627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.398924 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.398903 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ff297761-kserve-self-signed-certs\"" Jul 29 02:58:08.410960 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.410938 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn"] Jul 29 02:58:08.477478 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.477454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-model-cache\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.477588 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.477479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg577\" (UniqueName: \"kubernetes.io/projected/e61748b1-0ad5-4f93-8d95-62bbf19e502b-kube-api-access-dg577\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.477588 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.477501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tmp-dir\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.477588 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.477520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-home\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.477760 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.477612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tls-certs\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.477760 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.477676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-dshm\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.578877 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.578849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tls-certs\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579054 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.578885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-dshm\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579054 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.578953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-model-cache\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579054 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.578978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg577\" (UniqueName: \"kubernetes.io/projected/e61748b1-0ad5-4f93-8d95-62bbf19e502b-kube-api-access-dg577\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579054 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.579007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tmp-dir\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579250 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.579063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-home\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579423 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.579401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-home\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579554 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.579401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-model-cache\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.579616 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.579566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tmp-dir\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.581664 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.581619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-dshm\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.581784 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.581722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tls-certs\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.586202 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.586173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg577\" (UniqueName: \"kubernetes.io/projected/e61748b1-0ad5-4f93-8d95-62bbf19e502b-kube-api-access-dg577\") pod \"e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.706872 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.706843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:08.836622 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:08.836600 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn"] Jul 29 02:58:08.838756 ip-10-0-141-222 kubenswrapper[2579]: W0729 02:58:08.838716 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode61748b1_0ad5_4f93_8d95_62bbf19e502b.slice/crio-36ca43be5caea778b6e6d9df3199fd9a8026926cf9e89c68feede0874b96ec2b WatchSource:0}: Error finding container 36ca43be5caea778b6e6d9df3199fd9a8026926cf9e89c68feede0874b96ec2b: Status 404 returned error can't find the container with id 36ca43be5caea778b6e6d9df3199fd9a8026926cf9e89c68feede0874b96ec2b Jul 29 02:58:09.560766 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:09.560735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" event={"ID":"e61748b1-0ad5-4f93-8d95-62bbf19e502b","Type":"ContainerStarted","Data":"e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0"} Jul 29 02:58:09.561180 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:09.560774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" event={"ID":"e61748b1-0ad5-4f93-8d95-62bbf19e502b","Type":"ContainerStarted","Data":"36ca43be5caea778b6e6d9df3199fd9a8026926cf9e89c68feede0874b96ec2b"} Jul 29 02:58:09.576723 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:09.576682 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" podStartSLOduration=1.576666463 podStartE2EDuration="1.576666463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 02:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 02:58:09.574545544 +0000 UTC m=+1379.635020952" watchObservedRunningTime="2026-07-29 02:58:09.576666463 +0000 UTC m=+1379.637141863" Jul 29 02:58:18.707918 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:18.707889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:18.708279 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:18.707954 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:18.719970 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:18.719947 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:19.621061 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:19.621036 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:58:47.694528 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:58:47.694485 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn"] Jul 29 02:59:02.712845 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:02.712805 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" podUID="e61748b1-0ad5-4f93-8d95-62bbf19e502b" containerName="main" containerID="cri-o://e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0" gracePeriod=45 Jul 29 02:59:02.956553 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:02.956523 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:59:03.103851 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.103781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-home\") pod \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " Jul 29 02:59:03.103966 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.103866 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-model-cache\") pod \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " Jul 29 02:59:03.103966 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.103904 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tmp-dir\") pod \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " Jul 29 02:59:03.103966 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.103942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-dshm\") pod \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " Jul 29 02:59:03.104122 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.103975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg577\" (UniqueName: \"kubernetes.io/projected/e61748b1-0ad5-4f93-8d95-62bbf19e502b-kube-api-access-dg577\") pod \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " Jul 29 02:59:03.104122 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.103997 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tls-certs\") pod \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\" (UID: \"e61748b1-0ad5-4f93-8d95-62bbf19e502b\") " Jul 29 02:59:03.104122 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.104052 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-home" (OuterVolumeSpecName: "home") pod "e61748b1-0ad5-4f93-8d95-62bbf19e502b" (UID: "e61748b1-0ad5-4f93-8d95-62bbf19e502b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:59:03.104122 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.104100 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e61748b1-0ad5-4f93-8d95-62bbf19e502b" (UID: "e61748b1-0ad5-4f93-8d95-62bbf19e502b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:59:03.110571 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.104518 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e61748b1-0ad5-4f93-8d95-62bbf19e502b" (UID: "e61748b1-0ad5-4f93-8d95-62bbf19e502b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:59:03.110571 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.104712 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-home\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:59:03.110571 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.104730 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-model-cache\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:59:03.110571 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.107034 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e61748b1-0ad5-4f93-8d95-62bbf19e502b" (UID: "e61748b1-0ad5-4f93-8d95-62bbf19e502b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 02:59:03.111623 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.111156 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-dshm" (OuterVolumeSpecName: "dshm") pod "e61748b1-0ad5-4f93-8d95-62bbf19e502b" (UID: "e61748b1-0ad5-4f93-8d95-62bbf19e502b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 02:59:03.112563 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.112534 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61748b1-0ad5-4f93-8d95-62bbf19e502b-kube-api-access-dg577" (OuterVolumeSpecName: "kube-api-access-dg577") pod "e61748b1-0ad5-4f93-8d95-62bbf19e502b" (UID: "e61748b1-0ad5-4f93-8d95-62bbf19e502b"). InnerVolumeSpecName "kube-api-access-dg577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 02:59:03.205350 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.205289 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg577\" (UniqueName: \"kubernetes.io/projected/e61748b1-0ad5-4f93-8d95-62bbf19e502b-kube-api-access-dg577\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:59:03.205350 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.205314 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tls-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:59:03.205350 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.205328 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-tmp-dir\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:59:03.205350 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.205340 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e61748b1-0ad5-4f93-8d95-62bbf19e502b-dshm\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 02:59:03.814923 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.814891 2579 generic.go:356] "Generic (PLEG): container finished" podID="e61748b1-0ad5-4f93-8d95-62bbf19e502b" containerID="e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0" exitCode=0 Jul 29 02:59:03.815365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.814949 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" Jul 29 02:59:03.815365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.814980 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" event={"ID":"e61748b1-0ad5-4f93-8d95-62bbf19e502b","Type":"ContainerDied","Data":"e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0"} Jul 29 02:59:03.815365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.815054 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn" event={"ID":"e61748b1-0ad5-4f93-8d95-62bbf19e502b","Type":"ContainerDied","Data":"36ca43be5caea778b6e6d9df3199fd9a8026926cf9e89c68feede0874b96ec2b"} Jul 29 02:59:03.815365 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.815081 2579 scope.go:117] "RemoveContainer" containerID="e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0" Jul 29 02:59:03.823724 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.823710 2579 scope.go:117] "RemoveContainer" containerID="e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0" Jul 29 02:59:03.823943 ip-10-0-141-222 kubenswrapper[2579]: E0729 02:59:03.823922 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0\": container with ID starting with e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0 not found: ID does not exist" containerID="e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0" Jul 29 02:59:03.823988 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.823954 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0"} err="failed to get container status \"e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0\": rpc error: code = NotFound desc = could not find container \"e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0\": container with ID starting with e83378c318c18a4d70082f9cf974090185fe9780d703387353ada3c3826d13f0 not found: ID does not exist" Jul 29 02:59:03.835348 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.835327 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn"] Jul 29 02:59:03.838898 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:03.838881 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ff297761-kserve-54bc8f866c-7wqwn"] Jul 29 02:59:04.453115 ip-10-0-141-222 kubenswrapper[2579]: I0729 02:59:04.453085 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61748b1-0ad5-4f93-8d95-62bbf19e502b" path="/var/lib/kubelet/pods/e61748b1-0ad5-4f93-8d95-62bbf19e502b/volumes" Jul 29 03:00:10.410505 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:00:10.410388 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:00:10.414428 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:00:10.412749 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:02:35.830971 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.830892 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw"] Jul 29 03:02:35.831511 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.831324 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61748b1-0ad5-4f93-8d95-62bbf19e502b" containerName="main" Jul 29 03:02:35.831511 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.831338 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61748b1-0ad5-4f93-8d95-62bbf19e502b" containerName="main" Jul 29 03:02:35.831511 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.831428 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61748b1-0ad5-4f93-8d95-62bbf19e502b" containerName="main" Jul 29 03:02:35.834369 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.834350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.836734 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.836712 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bf957620-openshift-default-dockercfg-n62s9\"" Jul 29 03:02:35.854706 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.854679 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw"] Jul 29 03:02:35.985085 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-socket\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985085 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-envoy\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985288 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-data\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985288 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-kube-api-access-2qzdg\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985288 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-credential-socket\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985288 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-podinfo\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985429 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istiod-ca-cert\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985429 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-certs\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:35.985429 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:35.985338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-token\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086373 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istiod-ca-cert\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086373 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-certs\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086373 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-token\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-socket\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-envoy\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-data\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-kube-api-access-2qzdg\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-credential-socket\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086893 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-certs\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086893 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086824 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-socket\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086995 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-data\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.086995 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.086937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-podinfo\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.087103 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.087005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-credential-socket\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.087321 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.087303 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istiod-ca-cert\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.088648 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.088623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-envoy\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.089180 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.089162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-podinfo\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.094604 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.094577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-token\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.095068 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.095046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-kube-api-access-2qzdg\") pod \"e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.144905 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.144884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:36.295589 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.295565 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw"] Jul 29 03:02:36.297746 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:02:36.297719 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f75a5c_ec7f_4d7f_9c10_a8e546e2efc4.slice/crio-0016a286a6310f0149cd6ad040aa78a2c915ccf89cd308363e695f1fe2d930a6 WatchSource:0}: Error finding container 0016a286a6310f0149cd6ad040aa78a2c915ccf89cd308363e695f1fe2d930a6: Status 404 returned error can't find the container with id 0016a286a6310f0149cd6ad040aa78a2c915ccf89cd308363e695f1fe2d930a6 Jul 29 03:02:36.299940 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.299908 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:02:36.300059 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.299985 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:02:36.300059 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.300042 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:02:36.784037 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.783982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" event={"ID":"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4","Type":"ContainerStarted","Data":"bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca"} Jul 29 03:02:36.784225 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.784052 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" event={"ID":"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4","Type":"ContainerStarted","Data":"0016a286a6310f0149cd6ad040aa78a2c915ccf89cd308363e695f1fe2d930a6"} Jul 29 03:02:36.803994 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:36.803937 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" podStartSLOduration=1.803918457 podStartE2EDuration="1.803918457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:02:36.80274773 +0000 UTC m=+1646.863223114" watchObservedRunningTime="2026-07-29 03:02:36.803918457 +0000 UTC m=+1646.864393844" Jul 29 03:02:37.146115 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:37.146035 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:37.150840 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:37.150819 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:37.791424 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:37.791397 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:37.792345 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:37.792323 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:38.427061 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.427030 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb"] Jul 29 03:02:38.430393 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.430372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.435083 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.435062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bf957620-serving-cert\"" Jul 29 03:02:38.441916 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.441881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb"] Jul 29 03:02:38.611205 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.611172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714a7283-459c-42c0-83b2-9c42f2e69b0b-openshift-service-ca\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.611353 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.611239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.611353 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.611266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57rsh\" (UniqueName: \"kubernetes.io/projected/714a7283-459c-42c0-83b2-9c42f2e69b0b-kube-api-access-57rsh\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.712153 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.712128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714a7283-459c-42c0-83b2-9c42f2e69b0b-openshift-service-ca\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.712310 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.712182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.712310 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.712211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57rsh\" (UniqueName: \"kubernetes.io/projected/714a7283-459c-42c0-83b2-9c42f2e69b0b-kube-api-access-57rsh\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.712389 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:02:38.712343 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-serving-cert: secret "maas-api-e2e-ait-bf957620-serving-cert" not found Jul 29 03:02:38.712427 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:02:38.712417 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls podName:714a7283-459c-42c0-83b2-9c42f2e69b0b nodeName:}" failed. No retries permitted until 2026-07-29 03:02:39.212394002 +0000 UTC m=+1649.272869369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls") pod "maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" (UID: "714a7283-459c-42c0-83b2-9c42f2e69b0b") : secret "maas-api-e2e-ait-bf957620-serving-cert" not found Jul 29 03:02:38.712802 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.712784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714a7283-459c-42c0-83b2-9c42f2e69b0b-openshift-service-ca\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:38.724631 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:38.724613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57rsh\" (UniqueName: \"kubernetes.io/projected/714a7283-459c-42c0-83b2-9c42f2e69b0b-kube-api-access-57rsh\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:39.217093 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:39.217062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:39.219449 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:39.219424 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls\") pod \"maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:39.342928 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:39.342892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:39.487238 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:39.487212 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb"] Jul 29 03:02:39.489167 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:02:39.489137 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714a7283_459c_42c0_83b2_9c42f2e69b0b.slice/crio-ed17135d89c1de0cda69e9642ea0ca447d820e87e6e623022c2e4098728d95c6 WatchSource:0}: Error finding container ed17135d89c1de0cda69e9642ea0ca447d820e87e6e623022c2e4098728d95c6: Status 404 returned error can't find the container with id ed17135d89c1de0cda69e9642ea0ca447d820e87e6e623022c2e4098728d95c6 Jul 29 03:02:39.490534 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:39.490516 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 03:02:39.801766 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:39.801675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" event={"ID":"714a7283-459c-42c0-83b2-9c42f2e69b0b","Type":"ContainerStarted","Data":"ed17135d89c1de0cda69e9642ea0ca447d820e87e6e623022c2e4098728d95c6"} Jul 29 03:02:40.808214 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:40.808179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" event={"ID":"714a7283-459c-42c0-83b2-9c42f2e69b0b","Type":"ContainerStarted","Data":"570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1"} Jul 29 03:02:40.808665 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:40.808279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:40.829216 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:40.829150 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" podStartSLOduration=2.407181636 podStartE2EDuration="2.829119416s" totalImagesPullingTime="421.93778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:02:40.827265675 +0000 UTC m=+1650.887741059" watchObservedRunningTime="2026-07-29 03:02:40.829119416 +0000 UTC m=+1650.889594799" Jul 29 03:02:46.817990 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:46.817956 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:02:51.863715 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:51.863680 2579 generic.go:356] "Generic (PLEG): container finished" podID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerID="bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca" exitCode=0 Jul 29 03:02:51.864263 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:51.863752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" event={"ID":"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4","Type":"ContainerDied","Data":"bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca"} Jul 29 03:02:51.864263 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:51.864158 2579 scope.go:117] "RemoveContainer" containerID="bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca" Jul 29 03:02:51.865237 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:51.865210 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:02:51.865340 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:51.865301 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:02:51.865379 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:51.865339 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:02:52.791580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:52.791532 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:52.873080 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:52.873040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" event={"ID":"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4","Type":"ContainerStarted","Data":"4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5"} Jul 29 03:02:53.145398 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:53.145292 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:53.146611 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:53.146577 2579 patch_prober.go:28] interesting pod/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 03:02:53.146730 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:53.146662 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 03:02:54.145606 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:54.145571 2579 patch_prober.go:28] interesting pod/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 03:02:54.145984 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:54.145636 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 03:02:55.145618 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:55.145583 2579 patch_prober.go:28] interesting pod/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 03:02:55.145999 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:55.145634 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 03:02:56.145637 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:56.145592 2579 patch_prober.go:28] interesting pod/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 03:02:56.146128 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:56.145694 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 03:02:57.149192 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:57.149162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:57.149616 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:57.149436 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:02:57.150119 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:02:57.150102 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:03:09.069411 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.069382 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb"] Jul 29 03:03:09.069833 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.069582 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" podUID="714a7283-459c-42c0-83b2-9c42f2e69b0b" containerName="maas-api" containerID="cri-o://570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1" gracePeriod=30 Jul 29 03:03:09.322322 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.322254 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:03:09.380646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.380616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714a7283-459c-42c0-83b2-9c42f2e69b0b-openshift-service-ca\") pod \"714a7283-459c-42c0-83b2-9c42f2e69b0b\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " Jul 29 03:03:09.380811 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.380661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57rsh\" (UniqueName: \"kubernetes.io/projected/714a7283-459c-42c0-83b2-9c42f2e69b0b-kube-api-access-57rsh\") pod \"714a7283-459c-42c0-83b2-9c42f2e69b0b\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " Jul 29 03:03:09.380811 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.380697 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls\") pod \"714a7283-459c-42c0-83b2-9c42f2e69b0b\" (UID: \"714a7283-459c-42c0-83b2-9c42f2e69b0b\") " Jul 29 03:03:09.381004 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.380974 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714a7283-459c-42c0-83b2-9c42f2e69b0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "714a7283-459c-42c0-83b2-9c42f2e69b0b" (UID: "714a7283-459c-42c0-83b2-9c42f2e69b0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:03:09.382804 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.382780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "714a7283-459c-42c0-83b2-9c42f2e69b0b" (UID: "714a7283-459c-42c0-83b2-9c42f2e69b0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:03:09.382897 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.382834 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714a7283-459c-42c0-83b2-9c42f2e69b0b-kube-api-access-57rsh" (OuterVolumeSpecName: "kube-api-access-57rsh") pod "714a7283-459c-42c0-83b2-9c42f2e69b0b" (UID: "714a7283-459c-42c0-83b2-9c42f2e69b0b"). InnerVolumeSpecName "kube-api-access-57rsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:03:09.431539 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.431510 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw"] Jul 29 03:03:09.431755 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.431706 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" containerID="cri-o://4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5" gracePeriod=30 Jul 29 03:03:09.482333 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.482290 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/714a7283-459c-42c0-83b2-9c42f2e69b0b-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:09.482333 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.482321 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57rsh\" (UniqueName: \"kubernetes.io/projected/714a7283-459c-42c0-83b2-9c42f2e69b0b-kube-api-access-57rsh\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:09.482333 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.482331 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/714a7283-459c-42c0-83b2-9c42f2e69b0b-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:09.951489 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.951452 2579 generic.go:356] "Generic (PLEG): container finished" podID="714a7283-459c-42c0-83b2-9c42f2e69b0b" containerID="570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1" exitCode=0 Jul 29 03:03:09.951954 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.951522 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" Jul 29 03:03:09.951954 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.951520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" event={"ID":"714a7283-459c-42c0-83b2-9c42f2e69b0b","Type":"ContainerDied","Data":"570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1"} Jul 29 03:03:09.951954 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.951645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb" event={"ID":"714a7283-459c-42c0-83b2-9c42f2e69b0b","Type":"ContainerDied","Data":"ed17135d89c1de0cda69e9642ea0ca447d820e87e6e623022c2e4098728d95c6"} Jul 29 03:03:09.951954 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.951669 2579 scope.go:117] "RemoveContainer" containerID="570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1" Jul 29 03:03:09.962886 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.962855 2579 scope.go:117] "RemoveContainer" containerID="570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1" Jul 29 03:03:09.963248 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:09.963223 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1\": container with ID starting with 570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1 not found: ID does not exist" containerID="570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1" Jul 29 03:03:09.963337 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.963262 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1"} err="failed to get container status \"570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1\": rpc error: code = NotFound desc = could not find container \"570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1\": container with ID starting with 570368af6166adf6dce732696edad6ae114ece8fc7d64d711d37773f9b06e9c1 not found: ID does not exist" Jul 29 03:03:09.981921 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.981899 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb"] Jul 29 03:03:09.998550 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:09.998529 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf957620-5467c9bfbb-kdkgb"] Jul 29 03:03:10.454775 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.454740 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714a7283-459c-42c0-83b2-9c42f2e69b0b" path="/var/lib/kubelet/pods/714a7283-459c-42c0-83b2-9c42f2e69b0b/volumes" Jul 29 03:03:10.481394 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.481361 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg"] Jul 29 03:03:10.481917 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.481902 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714a7283-459c-42c0-83b2-9c42f2e69b0b" containerName="maas-api" Jul 29 03:03:10.481958 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.481921 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="714a7283-459c-42c0-83b2-9c42f2e69b0b" containerName="maas-api" Jul 29 03:03:10.482033 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.482004 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="714a7283-459c-42c0-83b2-9c42f2e69b0b" containerName="maas-api" Jul 29 03:03:10.484826 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.484808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.487461 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.487437 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ab95c5e9-gw-openshift-default-dockercfg-p82gk\"" Jul 29 03:03:10.502090 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.502065 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg"] Jul 29 03:03:10.592978 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.592948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-credential-socket\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593164 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.592983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-data\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593164 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.593005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-certs\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593164 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.593122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-envoy\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593164 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.593156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-socket\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593329 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.593182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-token\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593329 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.593216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bqx\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-kube-api-access-l2bqx\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593329 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.593279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-podinfo\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.593329 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.593325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istiod-ca-cert\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694252 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-credential-socket\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694252 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-data\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-certs\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-envoy\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-socket\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-token\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bqx\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-kube-api-access-l2bqx\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-podinfo\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istiod-ca-cert\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694862 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-credential-socket\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694862 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-socket\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.694968 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.694882 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-data\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.695047 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.695025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-certs\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.695264 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.695243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istiod-ca-cert\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.696542 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.696514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-envoy\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.696818 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.696800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-podinfo\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.704207 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.704182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-token\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.704307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.704183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bqx\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-kube-api-access-l2bqx\") pod \"e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.795542 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.795461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:10.952929 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.952597 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg"] Jul 29 03:03:10.957354 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.957317 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:10.957452 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.957401 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:10.957452 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:10.957440 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:11.964817 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:11.964782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" event={"ID":"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba","Type":"ContainerStarted","Data":"f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c"} Jul 29 03:03:11.965216 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:11.964840 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" event={"ID":"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba","Type":"ContainerStarted","Data":"7075f4979079f6aec0ff56ed95b36321bb7f1df3a8132620dd55d382d36caa2e"} Jul 29 03:03:11.985965 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:11.985917 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" podStartSLOduration=1.9859035619999998 podStartE2EDuration="1.985903562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:03:11.984916122 +0000 UTC m=+1682.045391510" watchObservedRunningTime="2026-07-29 03:03:11.985903562 +0000 UTC m=+1682.046378947" Jul 29 03:03:12.152470 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:12.152437 2579 patch_prober.go:28] interesting pod/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 03:03:12.152650 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:12.152497 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:03:12.795770 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:12.795740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:12.800098 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:12.800070 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:12.969069 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:12.969040 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:12.969706 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:12.969679 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:13.719639 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.719607 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg"] Jul 29 03:03:13.722831 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.722806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.726542 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.726519 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ab95c5e9-serving-cert\"" Jul 29 03:03:13.734477 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.734456 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg"] Jul 29 03:03:13.823780 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.823753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpbt\" (UniqueName: \"kubernetes.io/projected/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-kube-api-access-rxpbt\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.823893 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.823831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.823893 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.823871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.924787 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.924756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.924952 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.924799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.924952 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.924852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpbt\" (UniqueName: \"kubernetes.io/projected/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-kube-api-access-rxpbt\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.925092 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:13.924979 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-serving-cert: secret "maas-api-e2e-migrate-ab95c5e9-serving-cert" not found Jul 29 03:03:13.925092 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:13.925058 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls podName:547bdb27-f198-4e2b-9693-23ccc9cb1b8c nodeName:}" failed. No retries permitted until 2026-07-29 03:03:14.425038367 +0000 UTC m=+1684.485513750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls") pod "maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" (UID: "547bdb27-f198-4e2b-9693-23ccc9cb1b8c") : secret "maas-api-e2e-migrate-ab95c5e9-serving-cert" not found Jul 29 03:03:13.925458 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.925439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:13.933533 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:13.933506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpbt\" (UniqueName: \"kubernetes.io/projected/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-kube-api-access-rxpbt\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:14.430426 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.430380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:14.432754 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.432733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls\") pod \"maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:14.636667 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.636641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:14.677555 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.677537 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:03:14.788099 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.788039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg"] Jul 29 03:03:14.790451 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:03:14.790419 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547bdb27_f198_4e2b_9693_23ccc9cb1b8c.slice/crio-a4cd7ce3eccaee2fb99ed6fc036fd45d6743bc945fbe54dd27b8816c336d2cf6 WatchSource:0}: Error finding container a4cd7ce3eccaee2fb99ed6fc036fd45d6743bc945fbe54dd27b8816c336d2cf6: Status 404 returned error can't find the container with id a4cd7ce3eccaee2fb99ed6fc036fd45d6743bc945fbe54dd27b8816c336d2cf6 Jul 29 03:03:14.835186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835165 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-envoy\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835208 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-token\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-certs\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835279 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-credential-socket\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835301 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-podinfo\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835517 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835337 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-socket\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835517 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835368 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-data\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835517 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835426 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-kube-api-access-2qzdg\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835517 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835455 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istiod-ca-cert\") pod \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\" (UID: \"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4\") " Jul 29 03:03:14.835706 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835550 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:14.835706 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835567 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:14.835812 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835760 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.835812 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835777 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.835913 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.835901 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:03:14.836609 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.836582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-data" (OuterVolumeSpecName: "istio-data") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:14.836671 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.836654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:14.837563 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.837535 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:03:14.837662 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.837605 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-kube-api-access-2qzdg" (OuterVolumeSpecName: "kube-api-access-2qzdg") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "kube-api-access-2qzdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:03:14.837720 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.837663 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:03:14.837720 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.837679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" (UID: "a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:14.936180 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.936160 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.936180 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.936179 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.936300 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.936189 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.936300 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.936199 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.936300 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.936207 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-kube-api-access-2qzdg\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.936300 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.936215 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.936300 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.936223 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:14.979235 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.979176 2579 generic.go:356] "Generic (PLEG): container finished" podID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerID="4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5" exitCode=0 Jul 29 03:03:14.979235 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.979228 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" Jul 29 03:03:14.979379 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.979270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" event={"ID":"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4","Type":"ContainerDied","Data":"4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5"} Jul 29 03:03:14.979379 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.979335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw" event={"ID":"a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4","Type":"ContainerDied","Data":"0016a286a6310f0149cd6ad040aa78a2c915ccf89cd308363e695f1fe2d930a6"} Jul 29 03:03:14.979379 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.979367 2579 scope.go:117] "RemoveContainer" containerID="4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5" Jul 29 03:03:14.980439 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.980337 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" event={"ID":"547bdb27-f198-4e2b-9693-23ccc9cb1b8c","Type":"ContainerStarted","Data":"a4cd7ce3eccaee2fb99ed6fc036fd45d6743bc945fbe54dd27b8816c336d2cf6"} Jul 29 03:03:14.988521 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.988506 2579 scope.go:117] "RemoveContainer" containerID="bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca" Jul 29 03:03:14.996281 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.996263 2579 scope.go:117] "RemoveContainer" containerID="4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5" Jul 29 03:03:14.996510 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:14.996492 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5\": container with ID starting with 4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5 not found: ID does not exist" containerID="4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5" Jul 29 03:03:14.996580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.996523 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5"} err="failed to get container status \"4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5\": rpc error: code = NotFound desc = could not find container \"4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5\": container with ID starting with 4cf215919b7d5bfe03569979946ed6d4a3780a1ca64b364c9961f532e41696e5 not found: ID does not exist" Jul 29 03:03:14.996580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.996546 2579 scope.go:117] "RemoveContainer" containerID="bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca" Jul 29 03:03:14.996791 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:14.996773 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca\": container with ID starting with bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca not found: ID does not exist" containerID="bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca" Jul 29 03:03:14.996831 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.996800 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca"} err="failed to get container status \"bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca\": rpc error: code = NotFound desc = could not find container \"bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca\": container with ID starting with bac9d9da8cfe21ced07b90247a0240a2d252dbb697795330482d8e6cca7488ca not found: ID does not exist" Jul 29 03:03:14.999828 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:14.999809 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw"] Jul 29 03:03:15.004585 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:15.004568 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bf957620-openshift-default-cdcb7466-gdhzw"] Jul 29 03:03:15.988307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:15.988271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" event={"ID":"547bdb27-f198-4e2b-9693-23ccc9cb1b8c","Type":"ContainerStarted","Data":"15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6"} Jul 29 03:03:15.988677 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:15.988373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:16.005842 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:16.005792 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" podStartSLOduration=2.66943915 podStartE2EDuration="3.005773561s" totalImagesPullingTime="336.334411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:03:16.004388534 +0000 UTC m=+1686.064863919" watchObservedRunningTime="2026-07-29 03:03:16.005773561 +0000 UTC m=+1686.066248949" Jul 29 03:03:16.452559 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:16.452521 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" path="/var/lib/kubelet/pods/a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4/volumes" Jul 29 03:03:21.999460 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:21.999434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:27.042661 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:27.042633 2579 generic.go:356] "Generic (PLEG): container finished" podID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerID="f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c" exitCode=0 Jul 29 03:03:27.043044 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:27.042675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" event={"ID":"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba","Type":"ContainerDied","Data":"f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c"} Jul 29 03:03:27.043044 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:27.043004 2579 scope.go:117] "RemoveContainer" containerID="f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c" Jul 29 03:03:27.043877 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:27.043854 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:27.043926 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:27.043915 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:27.043962 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:27.043946 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:27.969875 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:27.969838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:28.049757 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:28.049722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" event={"ID":"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba","Type":"ContainerStarted","Data":"81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264"} Jul 29 03:03:28.796252 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:28.796213 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:28.797157 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:28.797134 2579 patch_prober.go:28] interesting pod/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 03:03:28.797266 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:28.797176 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 03:03:29.796318 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:29.796286 2579 patch_prober.go:28] interesting pod/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 03:03:29.796781 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:29.796346 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 03:03:30.795919 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:30.795888 2579 patch_prober.go:28] interesting pod/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 03:03:30.796133 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:30.795953 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 03:03:31.799431 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:31.799403 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:31.799780 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:31.799640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:31.800351 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:31.800331 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:42.335223 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.335167 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg"] Jul 29 03:03:42.335678 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.335532 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" podUID="547bdb27-f198-4e2b-9693-23ccc9cb1b8c" containerName="maas-api" containerID="cri-o://15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6" gracePeriod=30 Jul 29 03:03:42.581886 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.581863 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:42.673372 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.673275 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxpbt\" (UniqueName: \"kubernetes.io/projected/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-kube-api-access-rxpbt\") pod \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " Jul 29 03:03:42.673536 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.673408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls\") pod \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " Jul 29 03:03:42.673536 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.673452 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-openshift-service-ca\") pod \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\" (UID: \"547bdb27-f198-4e2b-9693-23ccc9cb1b8c\") " Jul 29 03:03:42.673883 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.673852 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "547bdb27-f198-4e2b-9693-23ccc9cb1b8c" (UID: "547bdb27-f198-4e2b-9693-23ccc9cb1b8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:03:42.675459 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.675434 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "547bdb27-f198-4e2b-9693-23ccc9cb1b8c" (UID: "547bdb27-f198-4e2b-9693-23ccc9cb1b8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:03:42.675553 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.675436 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-kube-api-access-rxpbt" (OuterVolumeSpecName: "kube-api-access-rxpbt") pod "547bdb27-f198-4e2b-9693-23ccc9cb1b8c" (UID: "547bdb27-f198-4e2b-9693-23ccc9cb1b8c"). InnerVolumeSpecName "kube-api-access-rxpbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:03:42.774388 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.774348 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpbt\" (UniqueName: \"kubernetes.io/projected/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-kube-api-access-rxpbt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:42.774388 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.774390 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:42.774691 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.774405 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/547bdb27-f198-4e2b-9693-23ccc9cb1b8c-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:42.791701 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.791674 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg"] Jul 29 03:03:42.791900 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:42.791879 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" containerID="cri-o://81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264" gracePeriod=30 Jul 29 03:03:43.129615 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.129577 2579 generic.go:356] "Generic (PLEG): container finished" podID="547bdb27-f198-4e2b-9693-23ccc9cb1b8c" containerID="15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6" exitCode=0 Jul 29 03:03:43.129783 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.129623 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" Jul 29 03:03:43.129783 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.129661 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" event={"ID":"547bdb27-f198-4e2b-9693-23ccc9cb1b8c","Type":"ContainerDied","Data":"15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6"} Jul 29 03:03:43.129783 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.129723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg" event={"ID":"547bdb27-f198-4e2b-9693-23ccc9cb1b8c","Type":"ContainerDied","Data":"a4cd7ce3eccaee2fb99ed6fc036fd45d6743bc945fbe54dd27b8816c336d2cf6"} Jul 29 03:03:43.129783 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.129746 2579 scope.go:117] "RemoveContainer" containerID="15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6" Jul 29 03:03:43.138574 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.138558 2579 scope.go:117] "RemoveContainer" containerID="15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6" Jul 29 03:03:43.138812 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:43.138795 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6\": container with ID starting with 15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6 not found: ID does not exist" containerID="15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6" Jul 29 03:03:43.138867 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.138823 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6"} err="failed to get container status \"15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6\": rpc error: code = NotFound desc = could not find container \"15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6\": container with ID starting with 15a211b37682a79918e7253f55a0faac6eb97fb359f2e5c8c1d561e8b475d3f6 not found: ID does not exist" Jul 29 03:03:43.160055 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.160033 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg"] Jul 29 03:03:43.167097 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.167075 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab95c5e9-6ccdbdc5fc-dscjg"] Jul 29 03:03:43.609472 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609442 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr"] Jul 29 03:03:43.609820 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609803 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" Jul 29 03:03:43.609820 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609816 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" Jul 29 03:03:43.609890 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609827 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547bdb27-f198-4e2b-9693-23ccc9cb1b8c" containerName="maas-api" Jul 29 03:03:43.609890 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609833 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="547bdb27-f198-4e2b-9693-23ccc9cb1b8c" containerName="maas-api" Jul 29 03:03:43.609890 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609839 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" Jul 29 03:03:43.609890 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609845 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" Jul 29 03:03:43.610010 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609916 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="547bdb27-f198-4e2b-9693-23ccc9cb1b8c" containerName="maas-api" Jul 29 03:03:43.610010 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609924 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" Jul 29 03:03:43.610010 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.609930 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f75a5c-ec7f-4d7f-9c10-a8e546e2efc4" containerName="istio-proxy" Jul 29 03:03:43.613619 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.613600 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.615851 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.615834 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4a476faa-openshift-default-dockercfg-kvrvr\"" Jul 29 03:03:43.624593 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.624573 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr"] Jul 29 03:03:43.681596 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-data\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.681705 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-certs\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.681705 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-podinfo\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.681705 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-envoy\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.681877 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxnh\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-kube-api-access-qmxnh\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.681877 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-credential-socket\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.681877 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-token\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.682036 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istiod-ca-cert\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.682036 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.681954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-socket\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783341 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-data\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783341 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-certs\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-podinfo\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-envoy\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxnh\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-kube-api-access-qmxnh\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-credential-socket\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-token\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istiod-ca-cert\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-socket\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783894 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-certs\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.783894 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.783783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-data\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.784102 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.784081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-socket\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.784169 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.784126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-credential-socket\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.784364 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.784339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istiod-ca-cert\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.785859 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.785829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-envoy\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.785956 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.785923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-podinfo\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.791693 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.791666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-token\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.791778 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.791732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxnh\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-kube-api-access-qmxnh\") pod \"e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:43.923581 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:43.923532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:44.060233 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.060205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr"] Jul 29 03:03:44.062129 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:03:44.062098 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a1ad25_c7d5_4f8f_97f8_0a2a2e2a4b91.slice/crio-c328f11ffa8c6a268ebe06b89f64fdf7ea2daaa4085f75ffcbb8248254e4d04c WatchSource:0}: Error finding container c328f11ffa8c6a268ebe06b89f64fdf7ea2daaa4085f75ffcbb8248254e4d04c: Status 404 returned error can't find the container with id c328f11ffa8c6a268ebe06b89f64fdf7ea2daaa4085f75ffcbb8248254e4d04c Jul 29 03:03:44.064467 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.064431 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:44.064577 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.064512 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:44.064577 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.064554 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:03:44.136272 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.136249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" event={"ID":"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91","Type":"ContainerStarted","Data":"921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518"} Jul 29 03:03:44.136367 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.136292 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" event={"ID":"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91","Type":"ContainerStarted","Data":"c328f11ffa8c6a268ebe06b89f64fdf7ea2daaa4085f75ffcbb8248254e4d04c"} Jul 29 03:03:44.155822 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.155768 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" podStartSLOduration=1.155745577 podStartE2EDuration="1.155745577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:03:44.153142581 +0000 UTC m=+1714.213617967" watchObservedRunningTime="2026-07-29 03:03:44.155745577 +0000 UTC m=+1714.216220963" Jul 29 03:03:44.452570 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.452541 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547bdb27-f198-4e2b-9693-23ccc9cb1b8c" path="/var/lib/kubelet/pods/547bdb27-f198-4e2b-9693-23ccc9cb1b8c/volumes" Jul 29 03:03:44.924560 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:44.924489 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:45.582477 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.582447 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5"] Jul 29 03:03:45.585982 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.585958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.588155 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.588132 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4a476faa-serving-cert\"" Jul 29 03:03:45.595607 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.595585 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5"] Jul 29 03:03:45.700138 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.700102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqvt\" (UniqueName: \"kubernetes.io/projected/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-kube-api-access-snqvt\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.700486 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.700467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-openshift-service-ca\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.700649 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.700632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.801884 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.801845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqvt\" (UniqueName: \"kubernetes.io/projected/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-kube-api-access-snqvt\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.802075 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.801900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-openshift-service-ca\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.802075 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.801934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.802210 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:45.802106 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-serving-cert: secret "maas-api-e2e-ait-4a476faa-serving-cert" not found Jul 29 03:03:45.802210 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:45.802178 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls podName:b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2 nodeName:}" failed. No retries permitted until 2026-07-29 03:03:46.302156921 +0000 UTC m=+1716.362632300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls") pod "maas-api-e2e-ait-4a476faa-675448956c-mvlc5" (UID: "b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2") : secret "maas-api-e2e-ait-4a476faa-serving-cert" not found Jul 29 03:03:45.802490 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.802473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-openshift-service-ca\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.810868 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.810838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqvt\" (UniqueName: \"kubernetes.io/projected/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-kube-api-access-snqvt\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:45.929288 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:45.929267 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:46.146473 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.146443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:46.147542 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.147508 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:03:46.307400 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.307317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:46.309670 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.309649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls\") pod \"maas-api-e2e-ait-4a476faa-675448956c-mvlc5\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:46.501111 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.501079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:46.802074 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.802042 2579 patch_prober.go:28] interesting pod/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 03:03:46.802222 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.802097 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:03:46.837861 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:46.837837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5"] Jul 29 03:03:46.839734 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:03:46.839715 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81cbbc2_3b9d_4cf4_af7f_becfd8d30ed2.slice/crio-762bdf987e1c6bd5862eafd915e4a18420a77697df68d581e98fc17a6fb03418 WatchSource:0}: Error finding container 762bdf987e1c6bd5862eafd915e4a18420a77697df68d581e98fc17a6fb03418: Status 404 returned error can't find the container with id 762bdf987e1c6bd5862eafd915e4a18420a77697df68d581e98fc17a6fb03418 Jul 29 03:03:47.151905 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:47.151819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" event={"ID":"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2","Type":"ContainerStarted","Data":"762bdf987e1c6bd5862eafd915e4a18420a77697df68d581e98fc17a6fb03418"} Jul 29 03:03:48.044945 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.044919 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:48.123939 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.123910 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-credential-socket\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124081 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.123950 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-socket\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124081 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.123973 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-certs\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124081 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-token\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124081 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124056 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2bqx\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-kube-api-access-l2bqx\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124097 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-envoy\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124132 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-podinfo\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124156 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istiod-ca-cert\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124185 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-data\") pod \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\" (UID: \"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba\") " Jul 29 03:03:48.124308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124199 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:48.124308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124258 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:48.124608 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124430 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.124608 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124449 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.124608 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:48.124725 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.124633 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:03:48.125266 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.125242 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:48.126213 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.126194 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:03:48.126409 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.126390 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:03:48.126587 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.126558 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-kube-api-access-l2bqx" (OuterVolumeSpecName: "kube-api-access-l2bqx") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "kube-api-access-l2bqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:03:48.126587 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.126575 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" (UID: "9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:03:48.159727 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.159699 2579 generic.go:356] "Generic (PLEG): container finished" podID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerID="81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264" exitCode=0 Jul 29 03:03:48.160118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.159748 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" Jul 29 03:03:48.160118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.159777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" event={"ID":"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba","Type":"ContainerDied","Data":"81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264"} Jul 29 03:03:48.160118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.159848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg" event={"ID":"9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba","Type":"ContainerDied","Data":"7075f4979079f6aec0ff56ed95b36321bb7f1df3a8132620dd55d382d36caa2e"} Jul 29 03:03:48.160118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.159877 2579 scope.go:117] "RemoveContainer" containerID="81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264" Jul 29 03:03:48.161929 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.161866 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" event={"ID":"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2","Type":"ContainerStarted","Data":"bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea"} Jul 29 03:03:48.172081 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.171787 2579 scope.go:117] "RemoveContainer" containerID="f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c" Jul 29 03:03:48.179711 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.179697 2579 scope.go:117] "RemoveContainer" containerID="81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264" Jul 29 03:03:48.179975 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:48.179955 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264\": container with ID starting with 81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264 not found: ID does not exist" containerID="81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264" Jul 29 03:03:48.180060 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.179984 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264"} err="failed to get container status \"81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264\": rpc error: code = NotFound desc = could not find container \"81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264\": container with ID starting with 81330c8756d61c8227774e7969f5a3918e80a0d509c73172f745e29ecfe2a264 not found: ID does not exist" Jul 29 03:03:48.180060 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.180005 2579 scope.go:117] "RemoveContainer" containerID="f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c" Jul 29 03:03:48.180257 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:03:48.180241 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c\": container with ID starting with f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c not found: ID does not exist" containerID="f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c" Jul 29 03:03:48.180296 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.180263 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c"} err="failed to get container status \"f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c\": rpc error: code = NotFound desc = could not find container \"f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c\": container with ID starting with f4f2815524090affb7aecb939c78559d6e870c2e8339d77a03b306e631bfb23c not found: ID does not exist" Jul 29 03:03:48.182697 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.182661 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" podStartSLOduration=2.302563778 podStartE2EDuration="3.182647881s" totalImagesPullingTime="880.084103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:03:48.181882511 +0000 UTC m=+1718.242357892" watchObservedRunningTime="2026-07-29 03:03:48.182647881 +0000 UTC m=+1718.243123273" Jul 29 03:03:48.194607 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.194587 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg"] Jul 29 03:03:48.198773 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.198755 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ab95c5e9-gw-openshift-default-7cdd9977bd-7mtxg"] Jul 29 03:03:48.225322 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.225302 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.225421 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.225326 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.225421 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.225342 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.225421 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.225365 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.225421 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.225380 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.225421 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.225393 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bqx\" (UniqueName: \"kubernetes.io/projected/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-kube-api-access-l2bqx\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.225421 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.225405 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:03:48.453225 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:48.453197 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" path="/var/lib/kubelet/pods/9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba/volumes" Jul 29 03:03:49.168503 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:49.168476 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:03:55.179732 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:03:55.179700 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:04:00.223008 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:00.222974 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerID="921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518" exitCode=0 Jul 29 03:04:00.223397 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:00.223046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" event={"ID":"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91","Type":"ContainerDied","Data":"921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518"} Jul 29 03:04:00.223441 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:00.223425 2579 scope.go:117] "RemoveContainer" containerID="921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518" Jul 29 03:04:00.224386 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:00.224363 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:00.224452 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:00.224428 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:00.224502 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:00.224455 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:01.147176 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:01.147147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:04:01.229565 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:01.229529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" event={"ID":"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91","Type":"ContainerStarted","Data":"7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11"} Jul 29 03:04:01.924687 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:01.924651 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:04:01.925429 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:01.925402 2579 patch_prober.go:28] interesting pod/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 29 03:04:01.925555 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:01.925449 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 29 03:04:02.924222 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:02.924190 2579 patch_prober.go:28] interesting pod/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 29 03:04:02.924587 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:02.924246 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 29 03:04:03.924417 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:03.924382 2579 patch_prober.go:28] interesting pod/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 29 03:04:03.924867 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:03.924441 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 29 03:04:04.927528 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:04.927493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:04:04.927986 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:04.927887 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:04:04.928564 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:04.928545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:04:16.362944 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.362907 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5"] Jul 29 03:04:16.363462 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.363153 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" podUID="b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" containerName="maas-api" containerID="cri-o://bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea" gracePeriod=30 Jul 29 03:04:16.625377 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.625311 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:04:16.768799 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.768765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snqvt\" (UniqueName: \"kubernetes.io/projected/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-kube-api-access-snqvt\") pod \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " Jul 29 03:04:16.768974 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.768858 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls\") pod \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " Jul 29 03:04:16.768974 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.768880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-openshift-service-ca\") pod \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\" (UID: \"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2\") " Jul 29 03:04:16.769257 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.769230 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" (UID: "b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:04:16.770799 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.770774 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" (UID: "b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:04:16.771258 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.771235 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-kube-api-access-snqvt" (OuterVolumeSpecName: "kube-api-access-snqvt") pod "b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" (UID: "b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2"). InnerVolumeSpecName "kube-api-access-snqvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:04:16.869897 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.869870 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:16.869897 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.869895 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:16.870126 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:16.869905 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snqvt\" (UniqueName: \"kubernetes.io/projected/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2-kube-api-access-snqvt\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:17.309074 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.309040 2579 generic.go:356] "Generic (PLEG): container finished" podID="b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" containerID="bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea" exitCode=0 Jul 29 03:04:17.309074 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.309076 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" Jul 29 03:04:17.309272 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.309103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" event={"ID":"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2","Type":"ContainerDied","Data":"bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea"} Jul 29 03:04:17.309272 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.309160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5" event={"ID":"b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2","Type":"ContainerDied","Data":"762bdf987e1c6bd5862eafd915e4a18420a77697df68d581e98fc17a6fb03418"} Jul 29 03:04:17.309272 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.309184 2579 scope.go:117] "RemoveContainer" containerID="bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea" Jul 29 03:04:17.317949 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.317930 2579 scope.go:117] "RemoveContainer" containerID="bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea" Jul 29 03:04:17.318211 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:04:17.318193 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea\": container with ID starting with bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea not found: ID does not exist" containerID="bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea" Jul 29 03:04:17.318275 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.318220 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea"} err="failed to get container status \"bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea\": rpc error: code = NotFound desc = could not find container \"bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea\": container with ID starting with bf9b546111c321fbe89c2f80f51b18b9704d17ba3b745fb3ff809f049dac1dea not found: ID does not exist" Jul 29 03:04:17.328342 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.328318 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5"] Jul 29 03:04:17.332010 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.331986 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4a476faa-675448956c-mvlc5"] Jul 29 03:04:17.766660 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.766630 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr"] Jul 29 03:04:17.767064 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:17.766874 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" containerID="cri-o://7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11" gracePeriod=30 Jul 29 03:04:18.452419 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.452386 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" path="/var/lib/kubelet/pods/b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2/volumes" Jul 29 03:04:18.547020 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.546982 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt"] Jul 29 03:04:18.547350 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547338 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" Jul 29 03:04:18.547397 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547352 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" Jul 29 03:04:18.547397 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547361 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" Jul 29 03:04:18.547397 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547366 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" Jul 29 03:04:18.547397 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547386 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" containerName="maas-api" Jul 29 03:04:18.547397 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547392 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" containerName="maas-api" Jul 29 03:04:18.547542 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547444 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" Jul 29 03:04:18.547542 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.547454 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b81cbbc2-3b9d-4cf4-af7f-becfd8d30ed2" containerName="maas-api" Jul 29 03:04:18.551512 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.551488 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.553535 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.553516 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-92d01eb1-openshift-default-dockercfg-b59mb\"" Jul 29 03:04:18.567396 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.564912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt"] Jul 29 03:04:18.686462 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-certs\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.686462 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-envoy\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.686709 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-socket\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.686709 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istiod-ca-cert\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.686709 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-credential-socket\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.686709 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-podinfo\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.686709 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxcn5\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-kube-api-access-zxcn5\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.686709 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-token\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.687005 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.686721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-data\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788091 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.787999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istiod-ca-cert\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788091 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-credential-socket\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788091 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-podinfo\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788091 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxcn5\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-kube-api-access-zxcn5\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-token\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-data\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-certs\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-envoy\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-socket\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-credential-socket\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-certs\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-data\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.788663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-socket\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.789191 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.788768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istiod-ca-cert\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.790460 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.790432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-podinfo\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.790569 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.790492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-envoy\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.795142 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.795125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-token\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.795452 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.795432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxcn5\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-kube-api-access-zxcn5\") pod \"e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:18.874178 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:18.874157 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:19.012862 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.012832 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt"] Jul 29 03:04:19.015491 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:04:19.015459 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdea822b_edaa_41e3_8db7_cf7acfb415b1.slice/crio-98083654c85e611ae3c0c8ec7a71d22929006130765d9495403a699c58906ee9 WatchSource:0}: Error finding container 98083654c85e611ae3c0c8ec7a71d22929006130765d9495403a699c58906ee9: Status 404 returned error can't find the container with id 98083654c85e611ae3c0c8ec7a71d22929006130765d9495403a699c58906ee9 Jul 29 03:04:19.017865 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.017806 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:19.017992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.017897 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:19.017992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.017941 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:19.321538 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.321452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" event={"ID":"cdea822b-edaa-41e3-8db7-cf7acfb415b1","Type":"ContainerStarted","Data":"44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe"} Jul 29 03:04:19.321538 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.321506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" event={"ID":"cdea822b-edaa-41e3-8db7-cf7acfb415b1","Type":"ContainerStarted","Data":"98083654c85e611ae3c0c8ec7a71d22929006130765d9495403a699c58906ee9"} Jul 29 03:04:19.340872 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.340827 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" podStartSLOduration=1.3408108269999999 podStartE2EDuration="1.340810827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:04:19.338101512 +0000 UTC m=+1749.398576896" watchObservedRunningTime="2026-07-29 03:04:19.340810827 +0000 UTC m=+1749.401286212" Jul 29 03:04:19.875099 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.875069 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:19.930260 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.930225 2579 patch_prober.go:28] interesting pod/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 03:04:19.930388 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:19.930288 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:04:20.768115 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.768088 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq"] Jul 29 03:04:20.768694 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.768674 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0d80d6-76ae-4dbf-a28a-7c59eccbbdba" containerName="istio-proxy" Jul 29 03:04:20.772064 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.772045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.774094 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.774076 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-92d01eb1-serving-cert\"" Jul 29 03:04:20.779625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.779517 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq"] Jul 29 03:04:20.802436 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.802415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wns\" (UniqueName: \"kubernetes.io/projected/167c8fed-79a5-4190-b0b0-f54a351e986d-kube-api-access-m8wns\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.802526 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.802470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167c8fed-79a5-4190-b0b0-f54a351e986d-openshift-service-ca\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.802526 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.802497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.886835 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.886802 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:20.903509 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.903478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167c8fed-79a5-4190-b0b0-f54a351e986d-openshift-service-ca\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.903689 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.903518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.903689 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.903574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wns\" (UniqueName: \"kubernetes.io/projected/167c8fed-79a5-4190-b0b0-f54a351e986d-kube-api-access-m8wns\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.903776 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:04:20.903690 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-serving-cert: secret "maas-api-e2e-derive-92d01eb1-serving-cert" not found Jul 29 03:04:20.903776 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:04:20.903760 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls podName:167c8fed-79a5-4190-b0b0-f54a351e986d nodeName:}" failed. No retries permitted until 2026-07-29 03:04:21.403740609 +0000 UTC m=+1751.464216005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls") pod "maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" (UID: "167c8fed-79a5-4190-b0b0-f54a351e986d") : secret "maas-api-e2e-derive-92d01eb1-serving-cert" not found Jul 29 03:04:20.904173 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.904156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167c8fed-79a5-4190-b0b0-f54a351e986d-openshift-service-ca\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:20.911565 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:20.911541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wns\" (UniqueName: \"kubernetes.io/projected/167c8fed-79a5-4190-b0b0-f54a351e986d-kube-api-access-m8wns\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:21.331437 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:21.331402 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:21.332435 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:21.332418 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:21.407567 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:21.407538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:21.409854 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:21.409833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls\") pod \"maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:21.686001 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:21.685961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:21.816943 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:21.816919 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq"] Jul 29 03:04:21.819117 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:04:21.819087 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167c8fed_79a5_4190_b0b0_f54a351e986d.slice/crio-dfabbdddb3f6ae1281e7628c91c760b24a20c79f145f71811f4179b0c6563f90 WatchSource:0}: Error finding container dfabbdddb3f6ae1281e7628c91c760b24a20c79f145f71811f4179b0c6563f90: Status 404 returned error can't find the container with id dfabbdddb3f6ae1281e7628c91c760b24a20c79f145f71811f4179b0c6563f90 Jul 29 03:04:22.337250 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:22.337214 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" event={"ID":"167c8fed-79a5-4190-b0b0-f54a351e986d","Type":"ContainerStarted","Data":"14ef43bcd69b4247f90c4d9df9dba818f6d5afc7324ca2633a3d8df6b9b564db"} Jul 29 03:04:22.337653 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:22.337264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" event={"ID":"167c8fed-79a5-4190-b0b0-f54a351e986d","Type":"ContainerStarted","Data":"dfabbdddb3f6ae1281e7628c91c760b24a20c79f145f71811f4179b0c6563f90"} Jul 29 03:04:22.363920 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:22.363856 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" podStartSLOduration=2.008440527 podStartE2EDuration="2.363836891s" totalImagesPullingTime="355.396364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:04:22.360941792 +0000 UTC m=+1752.421417190" watchObservedRunningTime="2026-07-29 03:04:22.363836891 +0000 UTC m=+1752.424312280" Jul 29 03:04:23.023547 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.023526 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:04:23.121265 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-socket\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121265 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-token\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121288 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmxnh\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-kube-api-access-qmxnh\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121337 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-credential-socket\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121372 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-certs\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121429 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-podinfo\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121454 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-envoy\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121796 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121478 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:23.121796 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121493 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istiod-ca-cert\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121796 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-data\") pod \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\" (UID: \"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91\") " Jul 29 03:04:23.121796 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121668 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:23.121796 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121690 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:23.122185 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121876 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.122185 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121910 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.122185 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121927 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.122534 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.121984 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:04:23.122622 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.122545 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:23.123625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.123601 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:04:23.123746 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.123723 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-kube-api-access-qmxnh" (OuterVolumeSpecName: "kube-api-access-qmxnh") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "kube-api-access-qmxnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:04:23.123837 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.123819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:04:23.123886 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.123861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" (UID: "a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:23.222896 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.222867 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.222896 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.222891 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.222896 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.222900 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.223122 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.222907 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.223122 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.222914 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.223122 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.222923 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxnh\" (UniqueName: \"kubernetes.io/projected/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91-kube-api-access-qmxnh\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:23.343613 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.343577 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerID="7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11" exitCode=0 Jul 29 03:04:23.344052 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.343628 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" Jul 29 03:04:23.344052 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.343639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" event={"ID":"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91","Type":"ContainerDied","Data":"7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11"} Jul 29 03:04:23.344052 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.343694 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr" event={"ID":"a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91","Type":"ContainerDied","Data":"c328f11ffa8c6a268ebe06b89f64fdf7ea2daaa4085f75ffcbb8248254e4d04c"} Jul 29 03:04:23.344052 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.343714 2579 scope.go:117] "RemoveContainer" containerID="7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11" Jul 29 03:04:23.344294 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.344275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:23.352509 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.352494 2579 scope.go:117] "RemoveContainer" containerID="921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518" Jul 29 03:04:23.359792 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.359771 2579 scope.go:117] "RemoveContainer" containerID="7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11" Jul 29 03:04:23.360050 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:04:23.360009 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11\": container with ID starting with 7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11 not found: ID does not exist" containerID="7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11" Jul 29 03:04:23.360109 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.360059 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11"} err="failed to get container status \"7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11\": rpc error: code = NotFound desc = could not find container \"7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11\": container with ID starting with 7129e83eb2f68a95984c62c6c5c19d7f77d415bd94a7a806d59525cdd4cd0b11 not found: ID does not exist" Jul 29 03:04:23.360109 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.360078 2579 scope.go:117] "RemoveContainer" containerID="921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518" Jul 29 03:04:23.360295 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:04:23.360277 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518\": container with ID starting with 921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518 not found: ID does not exist" containerID="921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518" Jul 29 03:04:23.360331 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.360303 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518"} err="failed to get container status \"921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518\": rpc error: code = NotFound desc = could not find container \"921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518\": container with ID starting with 921724d7f6f88929e1486258e7ea65fe227c227e567f61b38f416fa3c6e84518 not found: ID does not exist" Jul 29 03:04:23.364864 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.364843 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr"] Jul 29 03:04:23.366975 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:23.366956 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4a476faa-openshift-default-77cf99f444-hcrqr"] Jul 29 03:04:24.452817 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:24.452784 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" path="/var/lib/kubelet/pods/a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91/volumes" Jul 29 03:04:29.361307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:29.360462 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:34.404715 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:34.404655 2579 generic.go:356] "Generic (PLEG): container finished" podID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerID="44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe" exitCode=0 Jul 29 03:04:34.405104 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:34.404730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" event={"ID":"cdea822b-edaa-41e3-8db7-cf7acfb415b1","Type":"ContainerDied","Data":"44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe"} Jul 29 03:04:34.405151 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:34.405130 2579 scope.go:117] "RemoveContainer" containerID="44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe" Jul 29 03:04:34.406101 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:34.406068 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:34.406195 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:34.406141 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:34.406195 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:34.406172 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:04:35.411574 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:35.411540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" event={"ID":"cdea822b-edaa-41e3-8db7-cf7acfb415b1","Type":"ContainerStarted","Data":"2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0"} Jul 29 03:04:35.875135 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:35.875051 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:35.876345 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:35.876241 2579 patch_prober.go:28] interesting pod/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 03:04:35.876345 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:35.876309 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 03:04:36.331849 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:36.331807 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:36.874555 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:36.874518 2579 patch_prober.go:28] interesting pod/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 03:04:36.875166 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:36.875124 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 03:04:37.874596 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:37.874553 2579 patch_prober.go:28] interesting pod/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 03:04:37.875093 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:37.874618 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 03:04:38.874830 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:38.874789 2579 patch_prober.go:28] interesting pod/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 03:04:38.875231 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:38.874855 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 03:04:39.879045 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:39.879002 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:39.879891 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:39.879871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:51.333354 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.333313 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq"] Jul 29 03:04:51.333874 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.333573 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" podUID="167c8fed-79a5-4190-b0b0-f54a351e986d" containerName="maas-api" containerID="cri-o://14ef43bcd69b4247f90c4d9df9dba818f6d5afc7324ca2633a3d8df6b9b564db" gracePeriod=30 Jul 29 03:04:51.489963 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.489927 2579 generic.go:356] "Generic (PLEG): container finished" podID="167c8fed-79a5-4190-b0b0-f54a351e986d" containerID="14ef43bcd69b4247f90c4d9df9dba818f6d5afc7324ca2633a3d8df6b9b564db" exitCode=0 Jul 29 03:04:51.490126 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.489996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" event={"ID":"167c8fed-79a5-4190-b0b0-f54a351e986d","Type":"ContainerDied","Data":"14ef43bcd69b4247f90c4d9df9dba818f6d5afc7324ca2633a3d8df6b9b564db"} Jul 29 03:04:51.595328 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.595268 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:51.658193 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.658157 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt"] Jul 29 03:04:51.658464 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.658434 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" containerID="cri-o://2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0" gracePeriod=30 Jul 29 03:04:51.673728 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.673704 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167c8fed-79a5-4190-b0b0-f54a351e986d-openshift-service-ca\") pod \"167c8fed-79a5-4190-b0b0-f54a351e986d\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " Jul 29 03:04:51.673872 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.673807 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8wns\" (UniqueName: \"kubernetes.io/projected/167c8fed-79a5-4190-b0b0-f54a351e986d-kube-api-access-m8wns\") pod \"167c8fed-79a5-4190-b0b0-f54a351e986d\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " Jul 29 03:04:51.673872 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.673855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls\") pod \"167c8fed-79a5-4190-b0b0-f54a351e986d\" (UID: \"167c8fed-79a5-4190-b0b0-f54a351e986d\") " Jul 29 03:04:51.674127 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.674098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167c8fed-79a5-4190-b0b0-f54a351e986d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "167c8fed-79a5-4190-b0b0-f54a351e986d" (UID: "167c8fed-79a5-4190-b0b0-f54a351e986d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:04:51.676762 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.676739 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167c8fed-79a5-4190-b0b0-f54a351e986d-kube-api-access-m8wns" (OuterVolumeSpecName: "kube-api-access-m8wns") pod "167c8fed-79a5-4190-b0b0-f54a351e986d" (UID: "167c8fed-79a5-4190-b0b0-f54a351e986d"). InnerVolumeSpecName "kube-api-access-m8wns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:04:51.676913 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.676893 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "167c8fed-79a5-4190-b0b0-f54a351e986d" (UID: "167c8fed-79a5-4190-b0b0-f54a351e986d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:04:51.775156 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.775122 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wns\" (UniqueName: \"kubernetes.io/projected/167c8fed-79a5-4190-b0b0-f54a351e986d-kube-api-access-m8wns\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:51.775156 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.775153 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167c8fed-79a5-4190-b0b0-f54a351e986d-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:51.775332 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:51.775166 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167c8fed-79a5-4190-b0b0-f54a351e986d-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:52.495501 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:52.495470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" event={"ID":"167c8fed-79a5-4190-b0b0-f54a351e986d","Type":"ContainerDied","Data":"dfabbdddb3f6ae1281e7628c91c760b24a20c79f145f71811f4179b0c6563f90"} Jul 29 03:04:52.495850 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:52.495507 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq" Jul 29 03:04:52.495850 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:52.495539 2579 scope.go:117] "RemoveContainer" containerID="14ef43bcd69b4247f90c4d9df9dba818f6d5afc7324ca2633a3d8df6b9b564db" Jul 29 03:04:52.518643 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:52.518619 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq"] Jul 29 03:04:52.523131 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:52.523112 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92d01eb1-85d57448cb-49ghq"] Jul 29 03:04:54.452306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:54.452265 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="167c8fed-79a5-4190-b0b0-f54a351e986d" path="/var/lib/kubelet/pods/167c8fed-79a5-4190-b0b0-f54a351e986d/volumes" Jul 29 03:04:54.881969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:54.881885 2579 patch_prober.go:28] interesting pod/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 03:04:54.881969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:54.881951 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:04:56.904615 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:56.904589 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:57.016111 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016083 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-podinfo\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016233 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016116 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-token\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016233 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016131 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxcn5\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-kube-api-access-zxcn5\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016233 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016203 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-envoy\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016368 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016247 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istiod-ca-cert\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016368 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016273 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-credential-socket\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016368 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016313 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-certs\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016368 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016334 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-data\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016368 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016359 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-socket\") pod \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\" (UID: \"cdea822b-edaa-41e3-8db7-cf7acfb415b1\") " Jul 29 03:04:57.016669 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016548 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:57.016723 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016679 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.016776 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:57.016776 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016754 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:04:57.016776 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.016765 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:57.017630 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.017604 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:57.018328 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.018309 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:04:57.018328 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.018314 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:04:57.018457 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.018435 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-kube-api-access-zxcn5" (OuterVolumeSpecName: "kube-api-access-zxcn5") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "kube-api-access-zxcn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:04:57.018496 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.018477 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cdea822b-edaa-41e3-8db7-cf7acfb415b1" (UID: "cdea822b-edaa-41e3-8db7-cf7acfb415b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:04:57.117862 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117839 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.117862 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117861 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.117992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117870 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxcn5\" (UniqueName: \"kubernetes.io/projected/cdea822b-edaa-41e3-8db7-cf7acfb415b1-kube-api-access-zxcn5\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.117992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117879 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.117992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117887 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.117992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117895 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.117992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117903 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.117992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.117910 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdea822b-edaa-41e3-8db7-cf7acfb415b1-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:04:57.520964 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.520933 2579 generic.go:356] "Generic (PLEG): container finished" podID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerID="2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0" exitCode=0 Jul 29 03:04:57.521149 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.520974 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" Jul 29 03:04:57.521149 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.521037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" event={"ID":"cdea822b-edaa-41e3-8db7-cf7acfb415b1","Type":"ContainerDied","Data":"2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0"} Jul 29 03:04:57.521149 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.521090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt" event={"ID":"cdea822b-edaa-41e3-8db7-cf7acfb415b1","Type":"ContainerDied","Data":"98083654c85e611ae3c0c8ec7a71d22929006130765d9495403a699c58906ee9"} Jul 29 03:04:57.521149 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.521112 2579 scope.go:117] "RemoveContainer" containerID="2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0" Jul 29 03:04:57.529945 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.529931 2579 scope.go:117] "RemoveContainer" containerID="44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe" Jul 29 03:04:57.536989 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.536973 2579 scope.go:117] "RemoveContainer" containerID="2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0" Jul 29 03:04:57.537238 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:04:57.537223 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0\": container with ID starting with 2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0 not found: ID does not exist" containerID="2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0" Jul 29 03:04:57.537297 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.537245 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0"} err="failed to get container status \"2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0\": rpc error: code = NotFound desc = could not find container \"2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0\": container with ID starting with 2e997f08ba6fecd66b9059c7f83172d4289ea679b9f6a391063b082b22c48ff0 not found: ID does not exist" Jul 29 03:04:57.537297 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.537260 2579 scope.go:117] "RemoveContainer" containerID="44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe" Jul 29 03:04:57.537456 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:04:57.537442 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe\": container with ID starting with 44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe not found: ID does not exist" containerID="44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe" Jul 29 03:04:57.537499 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.537459 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe"} err="failed to get container status \"44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe\": rpc error: code = NotFound desc = could not find container \"44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe\": container with ID starting with 44c846968c92a8fc039b2bc2a1f940baa1a4320b308b998285c4502a0f120cfe not found: ID does not exist" Jul 29 03:04:57.541136 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.541116 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt"] Jul 29 03:04:57.545468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:57.545449 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-92d01eb1-openshift-default-6656f8dc84-4n5gt"] Jul 29 03:04:58.453029 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:04:58.452985 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" path="/var/lib/kubelet/pods/cdea822b-edaa-41e3-8db7-cf7acfb415b1/volumes" Jul 29 03:05:10.439700 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:05:10.439673 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:05:10.442481 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:05:10.442454 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:06:41.629523 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629487 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj"] Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629857 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629869 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629878 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629883 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629901 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167c8fed-79a5-4190-b0b0-f54a351e986d" containerName="maas-api" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629906 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="167c8fed-79a5-4190-b0b0-f54a351e986d" containerName="maas-api" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629912 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629917 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629973 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="167c8fed-79a5-4190-b0b0-f54a351e986d" containerName="maas-api" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629981 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629989 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" Jul 29 03:06:41.630118 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.629994 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdea822b-edaa-41e3-8db7-cf7acfb415b1" containerName="istio-proxy" Jul 29 03:06:41.633248 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.633231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.635622 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.635596 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3b7bc34b-openshift-default-dockercfg-45nbj\"" Jul 29 03:06:41.667929 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.667905 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj"] Jul 29 03:06:41.734694 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsn4g\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-kube-api-access-xsn4g\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.734858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-envoy\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.734858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-token\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.734858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-podinfo\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.734858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e3088-bf56-45e9-a021-b33264e0fe93-istiod-ca-cert\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.734858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-certs\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.735120 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-data\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.735120 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-socket\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.735120 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.734949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-credential-socket\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835406 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-envoy\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835406 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-token\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835628 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-podinfo\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835628 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e3088-bf56-45e9-a021-b33264e0fe93-istiod-ca-cert\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835628 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-certs\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835786 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-data\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835856 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-socket\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835944 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-credential-socket\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.835944 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835903 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-certs\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.836087 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsn4g\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-kube-api-access-xsn4g\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.836087 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.835999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-data\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.836219 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.836199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-credential-socket\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.836278 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.836241 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e3088-bf56-45e9-a021-b33264e0fe93-istiod-ca-cert\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.836518 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.836498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-socket\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.837744 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.837719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-envoy\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.838183 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.838164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-podinfo\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.849969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.849939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-token\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.850117 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.850096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsn4g\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-kube-api-access-xsn4g\") pod \"e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:41.942914 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:41.942887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:42.111583 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:42.111549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj"] Jul 29 03:06:42.114505 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:06:42.114480 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0e3088_bf56_45e9_a021_b33264e0fe93.slice/crio-040b2b16b11b5cb63d3f8a60f4503c4dc9f8d0044a18553d11f719465c63b70b WatchSource:0}: Error finding container 040b2b16b11b5cb63d3f8a60f4503c4dc9f8d0044a18553d11f719465c63b70b: Status 404 returned error can't find the container with id 040b2b16b11b5cb63d3f8a60f4503c4dc9f8d0044a18553d11f719465c63b70b Jul 29 03:06:42.116545 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:42.116517 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:06:42.116613 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:42.116575 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:06:42.116613 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:42.116602 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:06:43.000367 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:43.000326 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" event={"ID":"ff0e3088-bf56-45e9-a021-b33264e0fe93","Type":"ContainerStarted","Data":"0049c00c15a32ae80041a6aeb56229ae9692037e872e071e3c1b9cea5f44b2aa"} Jul 29 03:06:43.001113 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:43.000382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" event={"ID":"ff0e3088-bf56-45e9-a021-b33264e0fe93","Type":"ContainerStarted","Data":"040b2b16b11b5cb63d3f8a60f4503c4dc9f8d0044a18553d11f719465c63b70b"} Jul 29 03:06:43.031454 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:43.031406 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" podStartSLOduration=2.03139109 podStartE2EDuration="2.03139109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:06:43.02784102 +0000 UTC m=+1893.088316403" watchObservedRunningTime="2026-07-29 03:06:43.03139109 +0000 UTC m=+1893.091866475" Jul 29 03:06:43.943996 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:43.943965 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:43.948433 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:43.948410 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:44.005172 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:44.005133 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:44.005941 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:44.005924 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:45.272188 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.272155 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h"] Jul 29 03:06:45.272624 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.272601 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" Jul 29 03:06:45.272624 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.272625 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" Jul 29 03:06:45.272805 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.272757 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a1ad25-c7d5-4f8f-97f8-0a2a2e2a4b91" containerName="istio-proxy" Jul 29 03:06:45.276081 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.276063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.278627 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.278609 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3b7bc34b-serving-cert\"" Jul 29 03:06:45.292845 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.292826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h"] Jul 29 03:06:45.365143 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.365107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6b2\" (UniqueName: \"kubernetes.io/projected/0bed48e9-270e-4bcb-a07e-27b83291488b-kube-api-access-9j6b2\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.365318 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.365214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.365318 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.365302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bed48e9-270e-4bcb-a07e-27b83291488b-openshift-service-ca\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.465804 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.465762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.465987 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.465946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bed48e9-270e-4bcb-a07e-27b83291488b-openshift-service-ca\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.465987 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:06:45.465952 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-serving-cert: secret "maas-api-e2e-mt-3b7bc34b-serving-cert" not found Jul 29 03:06:45.466132 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.465999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6b2\" (UniqueName: \"kubernetes.io/projected/0bed48e9-270e-4bcb-a07e-27b83291488b-kube-api-access-9j6b2\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.466132 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:06:45.466045 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls podName:0bed48e9-270e-4bcb-a07e-27b83291488b nodeName:}" failed. No retries permitted until 2026-07-29 03:06:45.966005958 +0000 UTC m=+1896.026481326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls") pod "maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" (UID: "0bed48e9-270e-4bcb-a07e-27b83291488b") : secret "maas-api-e2e-mt-3b7bc34b-serving-cert" not found Jul 29 03:06:45.466866 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.466831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bed48e9-270e-4bcb-a07e-27b83291488b-openshift-service-ca\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.486443 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.486417 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6b2\" (UniqueName: \"kubernetes.io/projected/0bed48e9-270e-4bcb-a07e-27b83291488b-kube-api-access-9j6b2\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.969569 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.969534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:45.971870 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:45.971848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls\") pod \"maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:46.187540 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:46.187506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:46.346826 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:46.346792 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h"] Jul 29 03:06:46.348286 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:06:46.348251 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bed48e9_270e_4bcb_a07e_27b83291488b.slice/crio-a8359dca8ff0a0862acb346ed32e5dc56cbe1f6fc26d994b66d7694773a80517 WatchSource:0}: Error finding container a8359dca8ff0a0862acb346ed32e5dc56cbe1f6fc26d994b66d7694773a80517: Status 404 returned error can't find the container with id a8359dca8ff0a0862acb346ed32e5dc56cbe1f6fc26d994b66d7694773a80517 Jul 29 03:06:47.022193 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:47.022148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" event={"ID":"0bed48e9-270e-4bcb-a07e-27b83291488b","Type":"ContainerStarted","Data":"cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097"} Jul 29 03:06:47.022378 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:47.022216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" event={"ID":"0bed48e9-270e-4bcb-a07e-27b83291488b","Type":"ContainerStarted","Data":"a8359dca8ff0a0862acb346ed32e5dc56cbe1f6fc26d994b66d7694773a80517"} Jul 29 03:06:47.022378 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:47.022283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:47.068407 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:47.068292 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" podStartSLOduration=1.6022237430000001 podStartE2EDuration="2.068245263s" totalImagesPullingTime="466.02152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:06:47.067173192 +0000 UTC m=+1897.127648577" watchObservedRunningTime="2026-07-29 03:06:47.068245263 +0000 UTC m=+1897.128720649" Jul 29 03:06:53.032188 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:53.032160 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:06:58.076796 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:58.076764 2579 generic.go:356] "Generic (PLEG): container finished" podID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerID="0049c00c15a32ae80041a6aeb56229ae9692037e872e071e3c1b9cea5f44b2aa" exitCode=0 Jul 29 03:06:58.077320 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:58.076836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" event={"ID":"ff0e3088-bf56-45e9-a021-b33264e0fe93","Type":"ContainerDied","Data":"0049c00c15a32ae80041a6aeb56229ae9692037e872e071e3c1b9cea5f44b2aa"} Jul 29 03:06:58.077320 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:58.077301 2579 scope.go:117] "RemoveContainer" containerID="0049c00c15a32ae80041a6aeb56229ae9692037e872e071e3c1b9cea5f44b2aa" Jul 29 03:06:58.078275 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:58.078250 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:06:58.078364 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:58.078313 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:06:58.078364 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:58.078344 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:06:59.005376 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:59.005340 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:59.083682 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:59.083649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" event={"ID":"ff0e3088-bf56-45e9-a021-b33264e0fe93","Type":"ContainerStarted","Data":"514b46386489847cb9f4f1a633d444bd05f58fed9265262421b0442623cf85c4"} Jul 29 03:06:59.943707 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:59.943668 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:06:59.944855 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:59.944830 2579 patch_prober.go:28] interesting pod/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 29 03:06:59.944969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:06:59.944882 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 29 03:07:00.944124 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:00.944093 2579 patch_prober.go:28] interesting pod/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 29 03:07:00.944496 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:00.944145 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 29 03:07:01.944148 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:01.944109 2579 patch_prober.go:28] interesting pod/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 29 03:07:01.944585 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:01.944166 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 29 03:07:02.947260 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:02.947222 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:07:02.947723 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:02.947613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:07:02.948125 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:02.948107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:07:28.190000 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.189960 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h"] Jul 29 03:07:28.190598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.190253 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" podUID="0bed48e9-270e-4bcb-a07e-27b83291488b" containerName="maas-api" containerID="cri-o://cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097" gracePeriod=30 Jul 29 03:07:28.438831 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.438798 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:07:28.515048 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.514951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls\") pod \"0bed48e9-270e-4bcb-a07e-27b83291488b\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " Jul 29 03:07:28.515217 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.515072 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j6b2\" (UniqueName: \"kubernetes.io/projected/0bed48e9-270e-4bcb-a07e-27b83291488b-kube-api-access-9j6b2\") pod \"0bed48e9-270e-4bcb-a07e-27b83291488b\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " Jul 29 03:07:28.515217 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.515173 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bed48e9-270e-4bcb-a07e-27b83291488b-openshift-service-ca\") pod \"0bed48e9-270e-4bcb-a07e-27b83291488b\" (UID: \"0bed48e9-270e-4bcb-a07e-27b83291488b\") " Jul 29 03:07:28.515903 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.515876 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bed48e9-270e-4bcb-a07e-27b83291488b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0bed48e9-270e-4bcb-a07e-27b83291488b" (UID: "0bed48e9-270e-4bcb-a07e-27b83291488b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:07:28.517765 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.517733 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0bed48e9-270e-4bcb-a07e-27b83291488b" (UID: "0bed48e9-270e-4bcb-a07e-27b83291488b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:07:28.517765 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.517738 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bed48e9-270e-4bcb-a07e-27b83291488b-kube-api-access-9j6b2" (OuterVolumeSpecName: "kube-api-access-9j6b2") pod "0bed48e9-270e-4bcb-a07e-27b83291488b" (UID: "0bed48e9-270e-4bcb-a07e-27b83291488b"). InnerVolumeSpecName "kube-api-access-9j6b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:07:28.616168 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.616123 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j6b2\" (UniqueName: \"kubernetes.io/projected/0bed48e9-270e-4bcb-a07e-27b83291488b-kube-api-access-9j6b2\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:28.616168 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.616157 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bed48e9-270e-4bcb-a07e-27b83291488b-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:28.616168 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:28.616173 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bed48e9-270e-4bcb-a07e-27b83291488b-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:29.221584 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.221552 2579 generic.go:356] "Generic (PLEG): container finished" podID="0bed48e9-270e-4bcb-a07e-27b83291488b" containerID="cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097" exitCode=0 Jul 29 03:07:29.222046 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.221600 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" Jul 29 03:07:29.222046 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.221619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" event={"ID":"0bed48e9-270e-4bcb-a07e-27b83291488b","Type":"ContainerDied","Data":"cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097"} Jul 29 03:07:29.222046 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.221660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h" event={"ID":"0bed48e9-270e-4bcb-a07e-27b83291488b","Type":"ContainerDied","Data":"a8359dca8ff0a0862acb346ed32e5dc56cbe1f6fc26d994b66d7694773a80517"} Jul 29 03:07:29.222046 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.221682 2579 scope.go:117] "RemoveContainer" containerID="cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097" Jul 29 03:07:29.231042 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.230993 2579 scope.go:117] "RemoveContainer" containerID="cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097" Jul 29 03:07:29.231318 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:07:29.231301 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097\": container with ID starting with cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097 not found: ID does not exist" containerID="cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097" Jul 29 03:07:29.231383 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.231327 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097"} err="failed to get container status \"cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097\": rpc error: code = NotFound desc = could not find container \"cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097\": container with ID starting with cd0289e1a403278b7cf6cc2b907c397b9c4defb8cc837f3f91f100ac4c1a1097 not found: ID does not exist" Jul 29 03:07:29.243700 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.243659 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h"] Jul 29 03:07:29.247179 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:29.247137 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b7bc34b-56b5f84fbb-cxt9h"] Jul 29 03:07:30.452150 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:30.452115 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bed48e9-270e-4bcb-a07e-27b83291488b" path="/var/lib/kubelet/pods/0bed48e9-270e-4bcb-a07e-27b83291488b/volumes" Jul 29 03:07:35.021833 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:35.021798 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj"] Jul 29 03:07:35.022290 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:35.022032 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" containerID="cri-o://514b46386489847cb9f4f1a633d444bd05f58fed9265262421b0442623cf85c4" gracePeriod=30 Jul 29 03:07:40.277995 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.277963 2579 generic.go:356] "Generic (PLEG): container finished" podID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerID="514b46386489847cb9f4f1a633d444bd05f58fed9265262421b0442623cf85c4" exitCode=0 Jul 29 03:07:40.278362 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.278041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" event={"ID":"ff0e3088-bf56-45e9-a021-b33264e0fe93","Type":"ContainerDied","Data":"514b46386489847cb9f4f1a633d444bd05f58fed9265262421b0442623cf85c4"} Jul 29 03:07:40.278362 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.278099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" event={"ID":"ff0e3088-bf56-45e9-a021-b33264e0fe93","Type":"ContainerDied","Data":"040b2b16b11b5cb63d3f8a60f4503c4dc9f8d0044a18553d11f719465c63b70b"} Jul 29 03:07:40.278362 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.278120 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="040b2b16b11b5cb63d3f8a60f4503c4dc9f8d0044a18553d11f719465c63b70b" Jul 29 03:07:40.278362 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.278138 2579 scope.go:117] "RemoveContainer" containerID="0049c00c15a32ae80041a6aeb56229ae9692037e872e071e3c1b9cea5f44b2aa" Jul 29 03:07:40.279106 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.279087 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:07:40.428400 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428363 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-credential-socket\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428412 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-envoy\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428448 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsn4g\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-kube-api-access-xsn4g\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428472 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-token\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428492 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-data\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-podinfo\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428543 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e3088-bf56-45e9-a021-b33264e0fe93-istiod-ca-cert\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428564 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-socket\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.428598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428592 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-certs\") pod \"ff0e3088-bf56-45e9-a021-b33264e0fe93\" (UID: \"ff0e3088-bf56-45e9-a021-b33264e0fe93\") " Jul 29 03:07:40.429009 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428691 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:07:40.429009 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.428863 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.429147 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.429083 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:07:40.429147 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.429105 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0e3088-bf56-45e9-a021-b33264e0fe93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:07:40.429313 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.429247 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:07:40.429401 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.429379 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:07:40.430783 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.430747 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-kube-api-access-xsn4g" (OuterVolumeSpecName: "kube-api-access-xsn4g") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "kube-api-access-xsn4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:07:40.430886 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.430796 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:07:40.431109 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.431084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:07:40.431109 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.431094 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff0e3088-bf56-45e9-a021-b33264e0fe93" (UID: "ff0e3088-bf56-45e9-a021-b33264e0fe93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:07:40.530197 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530156 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.530197 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530193 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsn4g\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-kube-api-access-xsn4g\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.530197 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530208 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.530455 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530220 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.530455 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530230 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0e3088-bf56-45e9-a021-b33264e0fe93-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.530455 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530240 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e3088-bf56-45e9-a021-b33264e0fe93-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.530455 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530250 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:40.530455 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:40.530260 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0e3088-bf56-45e9-a021-b33264e0fe93-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:07:41.284255 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:41.284225 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj" Jul 29 03:07:41.316938 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:41.316904 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj"] Jul 29 03:07:41.322592 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:41.322567 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3b7bc34b-openshift-default-5fd648bddd-xsbfj"] Jul 29 03:07:42.452369 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:42.452333 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" path="/var/lib/kubelet/pods/ff0e3088-bf56-45e9-a021-b33264e0fe93/volumes" Jul 29 03:07:54.622501 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.622456 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv"] Jul 29 03:07:54.623009 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.622992 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" Jul 29 03:07:54.623009 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.623027 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" Jul 29 03:07:54.623173 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.623057 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bed48e9-270e-4bcb-a07e-27b83291488b" containerName="maas-api" Jul 29 03:07:54.623173 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.623066 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bed48e9-270e-4bcb-a07e-27b83291488b" containerName="maas-api" Jul 29 03:07:54.623173 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.623094 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" Jul 29 03:07:54.623173 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.623103 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" Jul 29 03:07:54.623363 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.623183 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" Jul 29 03:07:54.623363 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.623194 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bed48e9-270e-4bcb-a07e-27b83291488b" containerName="maas-api" Jul 29 03:07:54.634752 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.634719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.636859 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.636835 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c659b417-openshift-default-dockercfg-m6kr2\"" Jul 29 03:07:54.637008 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.636856 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv"] Jul 29 03:07:54.646532 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-socket\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646664 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-podinfo\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646664 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-certs\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646664 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74dp\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-kube-api-access-z74dp\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646772 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-envoy\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646810 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-token\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646864 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-data\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646864 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istiod-ca-cert\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.646961 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.646883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-credential-socket\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747708 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74dp\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-kube-api-access-z74dp\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747922 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-envoy\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747922 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-token\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747922 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-data\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747922 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istiod-ca-cert\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747922 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-credential-socket\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747922 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-socket\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.747922 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-podinfo\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.748288 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.747936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-certs\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.748381 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.748333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-data\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.748473 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.748402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-credential-socket\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.748546 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.748523 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-socket\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.748645 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.748614 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-certs\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.748761 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.748675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istiod-ca-cert\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.750148 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.750128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-envoy\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.750490 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.750471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-podinfo\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.755649 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.755626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74dp\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-kube-api-access-z74dp\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.755649 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.755636 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-token\") pod \"e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:54.947991 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:54.947954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:55.099141 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.098990 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv"] Jul 29 03:07:55.101342 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:07:55.101316 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb7c3eb_7ae0_4a34_a64e_b0751dcef660.slice/crio-2dfeea77d876ecd93ba948d706e4415d6a12f0a0666fdd60cabe5c26c8d37b75 WatchSource:0}: Error finding container 2dfeea77d876ecd93ba948d706e4415d6a12f0a0666fdd60cabe5c26c8d37b75: Status 404 returned error can't find the container with id 2dfeea77d876ecd93ba948d706e4415d6a12f0a0666fdd60cabe5c26c8d37b75 Jul 29 03:07:55.103430 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.103398 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:07:55.103532 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.103482 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:07:55.103597 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.103530 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:07:55.349558 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.349468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" event={"ID":"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660","Type":"ContainerStarted","Data":"a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5"} Jul 29 03:07:55.349558 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.349531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" event={"ID":"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660","Type":"ContainerStarted","Data":"2dfeea77d876ecd93ba948d706e4415d6a12f0a0666fdd60cabe5c26c8d37b75"} Jul 29 03:07:55.368745 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.368682 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" podStartSLOduration=1.368660773 podStartE2EDuration="1.368660773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:07:55.367168151 +0000 UTC m=+1965.427643529" watchObservedRunningTime="2026-07-29 03:07:55.368660773 +0000 UTC m=+1965.429136156" Jul 29 03:07:55.948581 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:55.948534 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:56.954819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:56.954785 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:57.360149 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.360053 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:57.361114 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.361083 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:07:57.516127 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.516087 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7"] Jul 29 03:07:57.516775 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.516750 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff0e3088-bf56-45e9-a021-b33264e0fe93" containerName="istio-proxy" Jul 29 03:07:57.518861 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.518841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.520942 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.520923 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c659b417-serving-cert\"" Jul 29 03:07:57.531684 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.531658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7"] Jul 29 03:07:57.572470 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.572435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pqw\" (UniqueName: \"kubernetes.io/projected/c7523da5-34ef-419a-b5ca-cdcf0bb98034-kube-api-access-l8pqw\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.572661 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.572496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.572661 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.572589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7523da5-34ef-419a-b5ca-cdcf0bb98034-openshift-service-ca\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.673833 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.673734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pqw\" (UniqueName: \"kubernetes.io/projected/c7523da5-34ef-419a-b5ca-cdcf0bb98034-kube-api-access-l8pqw\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.673995 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.673833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.673995 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.673876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7523da5-34ef-419a-b5ca-cdcf0bb98034-openshift-service-ca\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.674138 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:07:57.673987 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-serving-cert: secret "maas-api-e2e-mt-c659b417-serving-cert" not found Jul 29 03:07:57.674138 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:07:57.674093 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls podName:c7523da5-34ef-419a-b5ca-cdcf0bb98034 nodeName:}" failed. No retries permitted until 2026-07-29 03:07:58.174071147 +0000 UTC m=+1968.234546527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls") pod "maas-api-e2e-mt-c659b417-6856bd774c-49jj7" (UID: "c7523da5-34ef-419a-b5ca-cdcf0bb98034") : secret "maas-api-e2e-mt-c659b417-serving-cert" not found Jul 29 03:07:57.674637 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.674619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7523da5-34ef-419a-b5ca-cdcf0bb98034-openshift-service-ca\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:57.683902 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:57.683873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pqw\" (UniqueName: \"kubernetes.io/projected/c7523da5-34ef-419a-b5ca-cdcf0bb98034-kube-api-access-l8pqw\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:58.177480 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:58.177449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:58.180211 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:58.180188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls\") pod \"maas-api-e2e-mt-c659b417-6856bd774c-49jj7\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:58.432826 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:58.432793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:58.588448 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:58.588376 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7"] Jul 29 03:07:58.590626 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:07:58.590594 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7523da5_34ef_419a_b5ca_cdcf0bb98034.slice/crio-55b9a18a9c2f534e0fcc87acc5142dc4122024ebf4b2687e8b3aa7d8ad709ccb WatchSource:0}: Error finding container 55b9a18a9c2f534e0fcc87acc5142dc4122024ebf4b2687e8b3aa7d8ad709ccb: Status 404 returned error can't find the container with id 55b9a18a9c2f534e0fcc87acc5142dc4122024ebf4b2687e8b3aa7d8ad709ccb Jul 29 03:07:58.592078 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:58.592060 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 03:07:59.373131 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:59.373023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" event={"ID":"c7523da5-34ef-419a-b5ca-cdcf0bb98034","Type":"ContainerStarted","Data":"fb622bd424fa7562ed47229a5fecab122b1bb1ce4f2187fadd44f58281df86ce"} Jul 29 03:07:59.373131 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:59.373085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" event={"ID":"c7523da5-34ef-419a-b5ca-cdcf0bb98034","Type":"ContainerStarted","Data":"55b9a18a9c2f534e0fcc87acc5142dc4122024ebf4b2687e8b3aa7d8ad709ccb"} Jul 29 03:07:59.373635 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:59.373186 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:07:59.389174 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:07:59.389119 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" podStartSLOduration=1.881727404 podStartE2EDuration="2.389100874s" totalImagesPullingTime="507.37347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:07:59.38831333 +0000 UTC m=+1969.448788716" watchObservedRunningTime="2026-07-29 03:07:59.389100874 +0000 UTC m=+1969.449576260" Jul 29 03:08:05.383780 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:05.383698 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:08:11.431648 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:11.431609 2579 generic.go:356] "Generic (PLEG): container finished" podID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerID="a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5" exitCode=0 Jul 29 03:08:11.432094 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:11.431680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" event={"ID":"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660","Type":"ContainerDied","Data":"a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5"} Jul 29 03:08:11.432144 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:11.432098 2579 scope.go:117] "RemoveContainer" containerID="a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5" Jul 29 03:08:11.433068 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:11.433045 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:08:11.433139 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:11.433127 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:08:11.433176 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:11.433164 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:08:12.360655 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:12.360605 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:08:12.438355 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:12.438314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" event={"ID":"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660","Type":"ContainerStarted","Data":"686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294"} Jul 29 03:08:12.949088 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:12.949039 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:08:12.950369 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:12.950344 2579 patch_prober.go:28] interesting pod/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 03:08:12.950487 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:12.950404 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 03:08:13.949327 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:13.949290 2579 patch_prober.go:28] interesting pod/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 03:08:13.949749 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:13.949348 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 03:08:14.949205 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:14.949161 2579 patch_prober.go:28] interesting pod/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 03:08:14.949401 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:14.949232 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 03:08:15.949078 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:15.949006 2579 patch_prober.go:28] interesting pod/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 03:08:15.949266 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:15.949096 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 03:08:16.953117 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:16.953079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:08:16.953556 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:16.953377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:08:16.954208 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:08:16.954187 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:09:22.595287 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.595242 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7"] Jul 29 03:09:22.595847 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.595501 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" podUID="c7523da5-34ef-419a-b5ca-cdcf0bb98034" containerName="maas-api" containerID="cri-o://fb622bd424fa7562ed47229a5fecab122b1bb1ce4f2187fadd44f58281df86ce" gracePeriod=30 Jul 29 03:09:22.786751 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.786705 2579 generic.go:356] "Generic (PLEG): container finished" podID="c7523da5-34ef-419a-b5ca-cdcf0bb98034" containerID="fb622bd424fa7562ed47229a5fecab122b1bb1ce4f2187fadd44f58281df86ce" exitCode=0 Jul 29 03:09:22.786902 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.786778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" event={"ID":"c7523da5-34ef-419a-b5ca-cdcf0bb98034","Type":"ContainerDied","Data":"fb622bd424fa7562ed47229a5fecab122b1bb1ce4f2187fadd44f58281df86ce"} Jul 29 03:09:22.849133 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.849048 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:09:22.868946 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.868900 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7523da5-34ef-419a-b5ca-cdcf0bb98034-openshift-service-ca\") pod \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " Jul 29 03:09:22.869164 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.869045 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls\") pod \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " Jul 29 03:09:22.869164 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.869097 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8pqw\" (UniqueName: \"kubernetes.io/projected/c7523da5-34ef-419a-b5ca-cdcf0bb98034-kube-api-access-l8pqw\") pod \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\" (UID: \"c7523da5-34ef-419a-b5ca-cdcf0bb98034\") " Jul 29 03:09:22.869541 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.869414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7523da5-34ef-419a-b5ca-cdcf0bb98034-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7523da5-34ef-419a-b5ca-cdcf0bb98034" (UID: "c7523da5-34ef-419a-b5ca-cdcf0bb98034"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:09:22.871991 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.871948 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7523da5-34ef-419a-b5ca-cdcf0bb98034-kube-api-access-l8pqw" (OuterVolumeSpecName: "kube-api-access-l8pqw") pod "c7523da5-34ef-419a-b5ca-cdcf0bb98034" (UID: "c7523da5-34ef-419a-b5ca-cdcf0bb98034"). InnerVolumeSpecName "kube-api-access-l8pqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:09:22.872141 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.872122 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7523da5-34ef-419a-b5ca-cdcf0bb98034" (UID: "c7523da5-34ef-419a-b5ca-cdcf0bb98034"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:09:22.969945 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.969910 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8pqw\" (UniqueName: \"kubernetes.io/projected/c7523da5-34ef-419a-b5ca-cdcf0bb98034-kube-api-access-l8pqw\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:09:22.969945 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.969943 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7523da5-34ef-419a-b5ca-cdcf0bb98034-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:09:22.970162 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:22.969956 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7523da5-34ef-419a-b5ca-cdcf0bb98034-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:09:23.792915 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:23.792881 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" Jul 29 03:09:23.793444 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:23.792920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7" event={"ID":"c7523da5-34ef-419a-b5ca-cdcf0bb98034","Type":"ContainerDied","Data":"55b9a18a9c2f534e0fcc87acc5142dc4122024ebf4b2687e8b3aa7d8ad709ccb"} Jul 29 03:09:23.793444 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:23.792981 2579 scope.go:117] "RemoveContainer" containerID="fb622bd424fa7562ed47229a5fecab122b1bb1ce4f2187fadd44f58281df86ce" Jul 29 03:09:23.816555 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:23.816522 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7"] Jul 29 03:09:23.820875 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:23.820848 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c659b417-6856bd774c-49jj7"] Jul 29 03:09:24.452326 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:09:24.452290 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7523da5-34ef-419a-b5ca-cdcf0bb98034" path="/var/lib/kubelet/pods/c7523da5-34ef-419a-b5ca-cdcf0bb98034/volumes" Jul 29 03:10:10.476160 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:10.476124 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:10:10.478123 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:10.478098 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:10:13.956938 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:13.956907 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv"] Jul 29 03:10:13.957415 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:13.957137 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" containerID="cri-o://686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294" gracePeriod=30 Jul 29 03:10:16.956620 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:16.956553 2579 patch_prober.go:28] interesting pod/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 03:10:16.957082 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:16.956687 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:10:19.225371 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.225298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:10:19.279056 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.278995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-data\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279056 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279052 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z74dp\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-kube-api-access-z74dp\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279080 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istiod-ca-cert\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279105 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-token\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279132 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-credential-socket\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279173 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-certs\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279205 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-podinfo\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279233 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-envoy\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279281 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-socket\") pod \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\" (UID: \"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660\") " Jul 29 03:10:19.279675 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279432 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:10:19.279675 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279525 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:10:19.279675 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279575 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.279675 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279625 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:10:19.279888 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:10:19.279888 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.279787 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:10:19.281573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.281545 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:10:19.281750 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.281731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-kube-api-access-z74dp" (OuterVolumeSpecName: "kube-api-access-z74dp") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "kube-api-access-z74dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:10:19.281819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.281748 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:10:19.281894 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.281874 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" (UID: "dbb7c3eb-7ae0-4a34-a64e-b0751dcef660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:10:19.380399 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380362 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.380399 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380392 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.380399 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380402 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.380640 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380410 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.380640 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380420 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z74dp\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-kube-api-access-z74dp\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.380640 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380429 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.380640 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380437 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:19.380640 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:19.380444 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:10:20.061969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.061932 2579 generic.go:356] "Generic (PLEG): container finished" podID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerID="686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294" exitCode=0 Jul 29 03:10:20.062177 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.062046 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" Jul 29 03:10:20.062177 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.062039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" event={"ID":"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660","Type":"ContainerDied","Data":"686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294"} Jul 29 03:10:20.062297 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.062189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv" event={"ID":"dbb7c3eb-7ae0-4a34-a64e-b0751dcef660","Type":"ContainerDied","Data":"2dfeea77d876ecd93ba948d706e4415d6a12f0a0666fdd60cabe5c26c8d37b75"} Jul 29 03:10:20.062297 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.062223 2579 scope.go:117] "RemoveContainer" containerID="686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294" Jul 29 03:10:20.071536 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.071509 2579 scope.go:117] "RemoveContainer" containerID="a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5" Jul 29 03:10:20.081596 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.081482 2579 scope.go:117] "RemoveContainer" containerID="686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294" Jul 29 03:10:20.081805 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:10:20.081785 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294\": container with ID starting with 686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294 not found: ID does not exist" containerID="686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294" Jul 29 03:10:20.081857 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.081814 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294"} err="failed to get container status \"686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294\": rpc error: code = NotFound desc = could not find container \"686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294\": container with ID starting with 686fe4d9d5e1f1247581ae1ceda6768c08e11a37b0e7a642c4012775b3d30294 not found: ID does not exist" Jul 29 03:10:20.081857 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.081831 2579 scope.go:117] "RemoveContainer" containerID="a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5" Jul 29 03:10:20.082109 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:10:20.082090 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5\": container with ID starting with a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5 not found: ID does not exist" containerID="a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5" Jul 29 03:10:20.082178 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.082118 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5"} err="failed to get container status \"a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5\": rpc error: code = NotFound desc = could not find container \"a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5\": container with ID starting with a42429329a2694ca21756be18049493413380827cd2daba4f187d08a93e282c5 not found: ID does not exist" Jul 29 03:10:20.086501 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.086475 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv"] Jul 29 03:10:20.090086 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.090067 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c659b417-openshift-default-5c7698df7b-ndpxv"] Jul 29 03:10:20.452167 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:10:20.452123 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" path="/var/lib/kubelet/pods/dbb7c3eb-7ae0-4a34-a64e-b0751dcef660/volumes" Jul 29 03:11:09.540773 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.540676 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w"] Jul 29 03:11:09.541290 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.541072 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7523da5-34ef-419a-b5ca-cdcf0bb98034" containerName="maas-api" Jul 29 03:11:09.541290 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.541084 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7523da5-34ef-419a-b5ca-cdcf0bb98034" containerName="maas-api" Jul 29 03:11:09.541290 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.541091 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" Jul 29 03:11:09.541290 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.541096 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" Jul 29 03:11:09.541290 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.541152 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" Jul 29 03:11:09.541290 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.541161 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7523da5-34ef-419a-b5ca-cdcf0bb98034" containerName="maas-api" Jul 29 03:11:09.544331 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.544311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.546353 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.546328 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e2d41c-openshift-default-dockercfg-pmnx9\"" Jul 29 03:11:09.557721 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.557628 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w"] Jul 29 03:11:09.628600 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.628561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4glj\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-kube-api-access-r4glj\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.628600 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.628600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-socket\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.628881 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.628717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-podinfo\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.628881 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.628746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istiod-ca-cert\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.628881 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.628790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-certs\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.629082 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.628926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-envoy\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.629082 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.628993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-token\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.629256 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.629233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-credential-socket\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.629369 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.629334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-data\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730484 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-data\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4glj\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-kube-api-access-r4glj\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-socket\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-podinfo\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istiod-ca-cert\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-certs\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-envoy\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.730696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-token\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.731084 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-credential-socket\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.731084 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.730872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-data\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.731084 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.731009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-socket\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.731246 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.731185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-certs\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.731327 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.731307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-credential-socket\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.731582 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.731551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istiod-ca-cert\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.732984 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.732962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-envoy\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.733136 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.733118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-podinfo\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.738754 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.738733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-token\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.739128 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.739112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4glj\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-kube-api-access-r4glj\") pod \"e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:09.859835 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:09.859748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:10.001193 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.001156 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w"] Jul 29 03:11:10.004277 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:11:10.004241 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9030af_9e8c_4527_9cd9_8ce1865fe07f.slice/crio-243388a01c70f39a75184a4c20e83e16e7442bd78a5913ef807dfeb1b5f6c535 WatchSource:0}: Error finding container 243388a01c70f39a75184a4c20e83e16e7442bd78a5913ef807dfeb1b5f6c535: Status 404 returned error can't find the container with id 243388a01c70f39a75184a4c20e83e16e7442bd78a5913ef807dfeb1b5f6c535 Jul 29 03:11:10.006682 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.006649 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:10.006803 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.006716 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:10.006803 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.006746 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:10.307527 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.307481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" event={"ID":"af9030af-9e8c-4527-9cd9-8ce1865fe07f","Type":"ContainerStarted","Data":"c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07"} Jul 29 03:11:10.307744 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.307551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" event={"ID":"af9030af-9e8c-4527-9cd9-8ce1865fe07f","Type":"ContainerStarted","Data":"243388a01c70f39a75184a4c20e83e16e7442bd78a5913ef807dfeb1b5f6c535"} Jul 29 03:11:10.352463 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.352404 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" podStartSLOduration=1.352386717 podStartE2EDuration="1.352386717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:11:10.351852463 +0000 UTC m=+2160.412327849" watchObservedRunningTime="2026-07-29 03:11:10.352386717 +0000 UTC m=+2160.412862101" Jul 29 03:11:10.861752 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:10.861643 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:11.865834 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:11.865802 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:12.318023 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.317982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:12.318894 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.318871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:12.518179 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.518142 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x"] Jul 29 03:11:12.518716 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.518701 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" Jul 29 03:11:12.518771 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.518719 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" Jul 29 03:11:12.518813 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.518802 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb7c3eb-7ae0-4a34-a64e-b0751dcef660" containerName="istio-proxy" Jul 29 03:11:12.523609 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.523587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.526276 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.526249 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e2d41c-serving-cert\"" Jul 29 03:11:12.540175 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.540148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x"] Jul 29 03:11:12.556515 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.556082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf796d-e626-44dd-aecb-6058d21db1fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.556515 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.556147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf796d-e626-44dd-aecb-6058d21db1fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.556515 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.556195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvks\" (UniqueName: \"kubernetes.io/projected/6ecf796d-e626-44dd-aecb-6058d21db1fe-kube-api-access-wlvks\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.657337 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.657231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf796d-e626-44dd-aecb-6058d21db1fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.657337 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.657296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf796d-e626-44dd-aecb-6058d21db1fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.657569 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.657337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvks\" (UniqueName: \"kubernetes.io/projected/6ecf796d-e626-44dd-aecb-6058d21db1fe-kube-api-access-wlvks\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.658041 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.657983 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf796d-e626-44dd-aecb-6058d21db1fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.659789 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.659769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf796d-e626-44dd-aecb-6058d21db1fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.666551 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.666526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvks\" (UniqueName: \"kubernetes.io/projected/6ecf796d-e626-44dd-aecb-6058d21db1fe-kube-api-access-wlvks\") pod \"maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.834823 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.834780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:12.973573 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:12.973549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x"] Jul 29 03:11:12.975863 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:11:12.975836 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecf796d_e626_44dd_aecb_6058d21db1fe.slice/crio-f49d8aee57f4dc5026f4055a64ae8bb632e785011da3469149164e885bbf525f WatchSource:0}: Error finding container f49d8aee57f4dc5026f4055a64ae8bb632e785011da3469149164e885bbf525f: Status 404 returned error can't find the container with id f49d8aee57f4dc5026f4055a64ae8bb632e785011da3469149164e885bbf525f Jul 29 03:11:13.325788 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:13.325746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" event={"ID":"6ecf796d-e626-44dd-aecb-6058d21db1fe","Type":"ContainerStarted","Data":"f49d8aee57f4dc5026f4055a64ae8bb632e785011da3469149164e885bbf525f"} Jul 29 03:11:14.331149 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:14.331106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" event={"ID":"6ecf796d-e626-44dd-aecb-6058d21db1fe","Type":"ContainerStarted","Data":"d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3"} Jul 29 03:11:14.350826 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:14.350762 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" podStartSLOduration=1.78154759 podStartE2EDuration="2.350745304s" totalImagesPullingTime="569.197714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:11:14.349311389 +0000 UTC m=+2164.409786775" watchObservedRunningTime="2026-07-29 03:11:14.350745304 +0000 UTC m=+2164.411220688" Jul 29 03:11:15.336083 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:15.336054 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:21.360361 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:21.360328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:11:26.402084 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:26.402050 2579 generic.go:356] "Generic (PLEG): container finished" podID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerID="c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07" exitCode=0 Jul 29 03:11:26.402511 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:26.402126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" event={"ID":"af9030af-9e8c-4527-9cd9-8ce1865fe07f","Type":"ContainerDied","Data":"c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07"} Jul 29 03:11:26.402585 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:26.402566 2579 scope.go:117] "RemoveContainer" containerID="c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07" Jul 29 03:11:26.403592 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:26.403567 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:26.403654 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:26.403642 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:26.403721 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:26.403708 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:27.318932 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:27.318880 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:27.407987 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:27.407953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" event={"ID":"af9030af-9e8c-4527-9cd9-8ce1865fe07f","Type":"ContainerStarted","Data":"e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b"} Jul 29 03:11:27.860477 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:27.860443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:27.861550 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:27.861517 2579 patch_prober.go:28] interesting pod/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 03:11:27.861700 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:27.861598 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 03:11:28.861191 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:28.861155 2579 patch_prober.go:28] interesting pod/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 03:11:28.861830 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:28.861781 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 03:11:29.860896 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:29.860852 2579 patch_prober.go:28] interesting pod/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 03:11:29.861252 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:29.861219 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 03:11:30.861005 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:30.860962 2579 patch_prober.go:28] interesting pod/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 03:11:30.861338 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:30.861301 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 03:11:31.864281 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:31.864253 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:31.864730 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:31.864524 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:31.865286 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:31.865271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:11:37.996243 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:37.996205 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf"] Jul 29 03:11:38.000161 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.000139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.002443 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.002421 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2f84e8-openshift-default-dockercfg-sm274\"" Jul 29 03:11:38.011955 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.011930 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf"] Jul 29 03:11:38.097885 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.097839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-socket\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.097893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-credential-socket\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.097931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrj7q\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-kube-api-access-hrj7q\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.097962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-data\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.098042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-certs\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.098095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020422d8-17e1-4816-8cd7-2c18e9698235-istio-podinfo\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098376 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.098119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020422d8-17e1-4816-8cd7-2c18e9698235-istiod-ca-cert\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098376 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.098171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-envoy\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.098376 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.098210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-istio-token\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.198916 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.198865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-envoy\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.198916 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.198906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-istio-token\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-socket\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-credential-socket\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrj7q\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-kube-api-access-hrj7q\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-data\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-certs\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020422d8-17e1-4816-8cd7-2c18e9698235-istio-podinfo\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020422d8-17e1-4816-8cd7-2c18e9698235-istiod-ca-cert\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199681 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-socket\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199681 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-credential-socket\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.199823 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-certs\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.200001 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.199981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020422d8-17e1-4816-8cd7-2c18e9698235-istiod-ca-cert\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.200136 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.200110 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-data\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.201346 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.201321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-envoy\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.201850 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.201831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020422d8-17e1-4816-8cd7-2c18e9698235-istio-podinfo\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.206609 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.206581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-istio-token\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.206974 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.206957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrj7q\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-kube-api-access-hrj7q\") pod \"e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.310685 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.310585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:38.454447 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.454350 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf"] Jul 29 03:11:38.457387 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:11:38.457352 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020422d8_17e1_4816_8cd7_2c18e9698235.slice/crio-16eb532042c7be34d05b08e36d7c6344627228aa1768aed00375fd1fd972e8c6 WatchSource:0}: Error finding container 16eb532042c7be34d05b08e36d7c6344627228aa1768aed00375fd1fd972e8c6: Status 404 returned error can't find the container with id 16eb532042c7be34d05b08e36d7c6344627228aa1768aed00375fd1fd972e8c6 Jul 29 03:11:38.459613 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.459571 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:38.459722 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.459654 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:38.459722 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.459699 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:38.465646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:38.465613 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" event={"ID":"020422d8-17e1-4816-8cd7-2c18e9698235","Type":"ContainerStarted","Data":"16eb532042c7be34d05b08e36d7c6344627228aa1768aed00375fd1fd972e8c6"} Jul 29 03:11:39.472121 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:39.472080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" event={"ID":"020422d8-17e1-4816-8cd7-2c18e9698235","Type":"ContainerStarted","Data":"ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f"} Jul 29 03:11:39.494430 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:39.494380 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podStartSLOduration=2.494363403 podStartE2EDuration="2.494363403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:11:39.493151961 +0000 UTC m=+2189.553627346" watchObservedRunningTime="2026-07-29 03:11:39.494363403 +0000 UTC m=+2189.554838787" Jul 29 03:11:40.311744 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:40.311697 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:40.316818 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:40.316774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:40.476849 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:40.476811 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:40.477871 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:40.477852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:41.609755 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.609717 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484"] Jul 29 03:11:41.614204 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.614173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.616420 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.616395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2f84e8-serving-cert\"" Jul 29 03:11:41.624775 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.624746 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484"] Jul 29 03:11:41.734040 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.733968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbgc\" (UniqueName: \"kubernetes.io/projected/2f297412-7c03-4c04-bc8d-b4bb81803ebc-kube-api-access-thbgc\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.734239 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.734069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f297412-7c03-4c04-bc8d-b4bb81803ebc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.734239 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.734144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.835089 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.835043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f297412-7c03-4c04-bc8d-b4bb81803ebc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.835311 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.835112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.835311 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.835209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbgc\" (UniqueName: \"kubernetes.io/projected/2f297412-7c03-4c04-bc8d-b4bb81803ebc-kube-api-access-thbgc\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.835311 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:11:41.835276 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-serving-cert: secret "maas-api-e2e-inf-b-2f84e8-serving-cert" not found Jul 29 03:11:41.835476 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:11:41.835349 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls podName:2f297412-7c03-4c04-bc8d-b4bb81803ebc nodeName:}" failed. No retries permitted until 2026-07-29 03:11:42.335327981 +0000 UTC m=+2192.395803348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls") pod "maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" (UID: "2f297412-7c03-4c04-bc8d-b4bb81803ebc") : secret "maas-api-e2e-inf-b-2f84e8-serving-cert" not found Jul 29 03:11:41.835849 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.835825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f297412-7c03-4c04-bc8d-b4bb81803ebc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:41.846407 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:41.846370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbgc\" (UniqueName: \"kubernetes.io/projected/2f297412-7c03-4c04-bc8d-b4bb81803ebc-kube-api-access-thbgc\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:42.340111 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:42.340076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:42.342580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:42.342551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls\") pod \"maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:42.528268 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:42.528229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:42.666249 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:42.666222 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484"] Jul 29 03:11:42.668871 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:11:42.668837 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f297412_7c03_4c04_bc8d_b4bb81803ebc.slice/crio-86ac843355cdf18deb4c29622bf1cbdc589e7dc407269a91e13ecaaf363904a9 WatchSource:0}: Error finding container 86ac843355cdf18deb4c29622bf1cbdc589e7dc407269a91e13ecaaf363904a9: Status 404 returned error can't find the container with id 86ac843355cdf18deb4c29622bf1cbdc589e7dc407269a91e13ecaaf363904a9 Jul 29 03:11:43.492317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:43.492285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" event={"ID":"2f297412-7c03-4c04-bc8d-b4bb81803ebc","Type":"ContainerStarted","Data":"f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821"} Jul 29 03:11:43.492498 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:43.492335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" event={"ID":"2f297412-7c03-4c04-bc8d-b4bb81803ebc","Type":"ContainerStarted","Data":"86ac843355cdf18deb4c29622bf1cbdc589e7dc407269a91e13ecaaf363904a9"} Jul 29 03:11:43.492498 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:43.492478 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:43.511040 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:43.510973 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" podStartSLOduration=2.143452744 podStartE2EDuration="2.510956899s" totalImagesPullingTime="367.504155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:11:43.508921479 +0000 UTC m=+2193.569396864" watchObservedRunningTime="2026-07-29 03:11:43.510956899 +0000 UTC m=+2193.571432292" Jul 29 03:11:49.503194 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:49.503155 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:11:56.448217 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.447854 2579 patch_prober.go:28] interesting pod/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:48560->10.133.0.62:15021: read: connection reset by peer" start-of-body= Jul 29 03:11:56.448217 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.447932 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:48560->10.133.0.62:15021: read: connection reset by peer" Jul 29 03:11:56.560485 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.560448 2579 generic.go:356] "Generic (PLEG): container finished" podID="020422d8-17e1-4816-8cd7-2c18e9698235" containerID="ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f" exitCode=0 Jul 29 03:11:56.560671 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.560518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" event={"ID":"020422d8-17e1-4816-8cd7-2c18e9698235","Type":"ContainerDied","Data":"ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f"} Jul 29 03:11:56.561071 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.561054 2579 scope.go:117] "RemoveContainer" containerID="ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f" Jul 29 03:11:56.562134 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.562107 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:56.562213 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.562200 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:56.562251 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:56.562231 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:11:57.570608 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:57.570570 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" event={"ID":"020422d8-17e1-4816-8cd7-2c18e9698235","Type":"ContainerStarted","Data":"b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9"} Jul 29 03:11:58.311042 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:58.310982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:11:58.311268 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:58.311243 2579 patch_prober.go:28] interesting pod/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 03:11:58.311324 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:58.311306 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 03:11:59.311761 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:59.311718 2579 patch_prober.go:28] interesting pod/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 03:11:59.312205 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:11:59.311783 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 03:12:00.312154 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:12:00.312109 2579 patch_prober.go:28] interesting pod/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 03:12:00.312640 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:12:00.312191 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 03:12:01.311404 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:12:01.311364 2579 patch_prober.go:28] interesting pod/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 03:12:01.311591 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:12:01.311423 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 03:12:02.315501 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:12:02.315471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:12:02.315951 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:12:02.315753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:12:02.316581 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:12:02.316565 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:13:10.622499 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:13:10.622459 2579 scope.go:117] "RemoveContainer" containerID="514b46386489847cb9f4f1a633d444bd05f58fed9265262421b0442623cf85c4" Jul 29 03:14:03.468027 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.467792 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x"] Jul 29 03:14:03.469182 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.468687 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" podUID="6ecf796d-e626-44dd-aecb-6058d21db1fe" containerName="maas-api" containerID="cri-o://d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3" gracePeriod=30 Jul 29 03:14:03.735460 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.735432 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:14:03.772643 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.772604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlvks\" (UniqueName: \"kubernetes.io/projected/6ecf796d-e626-44dd-aecb-6058d21db1fe-kube-api-access-wlvks\") pod \"6ecf796d-e626-44dd-aecb-6058d21db1fe\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " Jul 29 03:14:03.772809 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.772683 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf796d-e626-44dd-aecb-6058d21db1fe-maas-api-tls\") pod \"6ecf796d-e626-44dd-aecb-6058d21db1fe\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " Jul 29 03:14:03.772809 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.772752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf796d-e626-44dd-aecb-6058d21db1fe-openshift-service-ca\") pod \"6ecf796d-e626-44dd-aecb-6058d21db1fe\" (UID: \"6ecf796d-e626-44dd-aecb-6058d21db1fe\") " Jul 29 03:14:03.773151 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.773123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecf796d-e626-44dd-aecb-6058d21db1fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ecf796d-e626-44dd-aecb-6058d21db1fe" (UID: "6ecf796d-e626-44dd-aecb-6058d21db1fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:14:03.774812 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.774775 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecf796d-e626-44dd-aecb-6058d21db1fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ecf796d-e626-44dd-aecb-6058d21db1fe" (UID: "6ecf796d-e626-44dd-aecb-6058d21db1fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:14:03.774812 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.774780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecf796d-e626-44dd-aecb-6058d21db1fe-kube-api-access-wlvks" (OuterVolumeSpecName: "kube-api-access-wlvks") pod "6ecf796d-e626-44dd-aecb-6058d21db1fe" (UID: "6ecf796d-e626-44dd-aecb-6058d21db1fe"). InnerVolumeSpecName "kube-api-access-wlvks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:14:03.873785 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.873751 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvks\" (UniqueName: \"kubernetes.io/projected/6ecf796d-e626-44dd-aecb-6058d21db1fe-kube-api-access-wlvks\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:03.873785 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.873780 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecf796d-e626-44dd-aecb-6058d21db1fe-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:03.873785 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:03.873789 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecf796d-e626-44dd-aecb-6058d21db1fe-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:04.209399 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.209360 2579 generic.go:356] "Generic (PLEG): container finished" podID="6ecf796d-e626-44dd-aecb-6058d21db1fe" containerID="d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3" exitCode=0 Jul 29 03:14:04.209605 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.209416 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" Jul 29 03:14:04.209605 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.209456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" event={"ID":"6ecf796d-e626-44dd-aecb-6058d21db1fe","Type":"ContainerDied","Data":"d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3"} Jul 29 03:14:04.209605 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.209516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x" event={"ID":"6ecf796d-e626-44dd-aecb-6058d21db1fe","Type":"ContainerDied","Data":"f49d8aee57f4dc5026f4055a64ae8bb632e785011da3469149164e885bbf525f"} Jul 29 03:14:04.209605 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.209544 2579 scope.go:117] "RemoveContainer" containerID="d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3" Jul 29 03:14:04.219220 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.219201 2579 scope.go:117] "RemoveContainer" containerID="d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3" Jul 29 03:14:04.219481 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:04.219461 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3\": container with ID starting with d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3 not found: ID does not exist" containerID="d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3" Jul 29 03:14:04.219529 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.219490 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3"} err="failed to get container status \"d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3\": rpc error: code = NotFound desc = could not find container \"d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3\": container with ID starting with d08c0c5da1ced525f1b24784ecf1587d5c1b18b968a48f1f074dde91e9a93fc3 not found: ID does not exist" Jul 29 03:14:04.229408 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.229369 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x"] Jul 29 03:14:04.232455 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.232434 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2d41c-758df9f9c-kjj6x"] Jul 29 03:14:04.452769 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:04.452734 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ecf796d-e626-44dd-aecb-6058d21db1fe" path="/var/lib/kubelet/pods/6ecf796d-e626-44dd-aecb-6058d21db1fe/volumes" Jul 29 03:14:09.971214 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:09.971180 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w"] Jul 29 03:14:09.971676 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:09.971444 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" containerID="cri-o://e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b" gracePeriod=30 Jul 29 03:14:15.250213 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.250148 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:14:15.265471 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.265429 2579 generic.go:356] "Generic (PLEG): container finished" podID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerID="e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b" exitCode=0 Jul 29 03:14:15.265625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.265476 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" Jul 29 03:14:15.265625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.265510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" event={"ID":"af9030af-9e8c-4527-9cd9-8ce1865fe07f","Type":"ContainerDied","Data":"e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b"} Jul 29 03:14:15.265625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.265566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w" event={"ID":"af9030af-9e8c-4527-9cd9-8ce1865fe07f","Type":"ContainerDied","Data":"243388a01c70f39a75184a4c20e83e16e7442bd78a5913ef807dfeb1b5f6c535"} Jul 29 03:14:15.265625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.265590 2579 scope.go:117] "RemoveContainer" containerID="e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b" Jul 29 03:14:15.275965 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.275947 2579 scope.go:117] "RemoveContainer" containerID="c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07" Jul 29 03:14:15.283956 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.283936 2579 scope.go:117] "RemoveContainer" containerID="e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b" Jul 29 03:14:15.284291 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:15.284252 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b\": container with ID starting with e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b not found: ID does not exist" containerID="e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b" Jul 29 03:14:15.284373 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.284303 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b"} err="failed to get container status \"e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b\": rpc error: code = NotFound desc = could not find container \"e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b\": container with ID starting with e4556f1d3079dc07af8ac5fbbcade783d33522299529ced914451d9dffa6d67b not found: ID does not exist" Jul 29 03:14:15.284373 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.284329 2579 scope.go:117] "RemoveContainer" containerID="c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07" Jul 29 03:14:15.284588 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:15.284562 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07\": container with ID starting with c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07 not found: ID does not exist" containerID="c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07" Jul 29 03:14:15.284630 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.284590 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07"} err="failed to get container status \"c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07\": rpc error: code = NotFound desc = could not find container \"c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07\": container with ID starting with c11e9773f268cad51bc5a1f0df545e016cbd9afc0d8e85ca75ebd634c5488f07 not found: ID does not exist" Jul 29 03:14:15.388324 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388291 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-envoy\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388528 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388347 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4glj\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-kube-api-access-r4glj\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388528 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388363 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-podinfo\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388528 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388399 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istiod-ca-cert\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388528 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388420 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-token\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388528 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-data\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388808 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388540 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-socket\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388808 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-certs\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388808 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388597 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-credential-socket\") pod \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\" (UID: \"af9030af-9e8c-4527-9cd9-8ce1865fe07f\") " Jul 29 03:14:15.388966 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388873 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:14:15.389061 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.388985 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.389180 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.389156 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:15.389275 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.389253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:15.389374 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.389347 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-data" (OuterVolumeSpecName: "istio-data") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:15.389509 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.389485 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:15.390714 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.390689 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:15.390714 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.390704 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-kube-api-access-r4glj" (OuterVolumeSpecName: "kube-api-access-r4glj") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "kube-api-access-r4glj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:14:15.390835 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.390753 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-token" (OuterVolumeSpecName: "istio-token") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:14:15.390835 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.390758 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af9030af-9e8c-4527-9cd9-8ce1865fe07f" (UID: "af9030af-9e8c-4527-9cd9-8ce1865fe07f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:14:15.490490 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490455 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4glj\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-kube-api-access-r4glj\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.490490 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490483 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.490490 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490493 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.490490 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490501 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.490753 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490509 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.490753 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490517 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.490753 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490525 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.490753 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.490532 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9030af-9e8c-4527-9cd9-8ce1865fe07f-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:15.587714 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.587680 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w"] Jul 29 03:14:15.591679 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:15.591656 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e2d41c-openshift-default-767547d84d-scw9w"] Jul 29 03:14:16.452688 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:16.452651 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" path="/var/lib/kubelet/pods/af9030af-9e8c-4527-9cd9-8ce1865fe07f/volumes" Jul 29 03:14:20.798464 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:20.798431 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484"] Jul 29 03:14:20.799349 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:20.799286 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" podUID="2f297412-7c03-4c04-bc8d-b4bb81803ebc" containerName="maas-api" containerID="cri-o://f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821" gracePeriod=30 Jul 29 03:14:21.247689 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.247663 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:14:21.296224 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.296182 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f297412-7c03-4c04-bc8d-b4bb81803ebc" containerID="f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821" exitCode=0 Jul 29 03:14:21.296224 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.296215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" event={"ID":"2f297412-7c03-4c04-bc8d-b4bb81803ebc","Type":"ContainerDied","Data":"f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821"} Jul 29 03:14:21.296224 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.296228 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" Jul 29 03:14:21.296476 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.296275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484" event={"ID":"2f297412-7c03-4c04-bc8d-b4bb81803ebc","Type":"ContainerDied","Data":"86ac843355cdf18deb4c29622bf1cbdc589e7dc407269a91e13ecaaf363904a9"} Jul 29 03:14:21.296476 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.296299 2579 scope.go:117] "RemoveContainer" containerID="f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821" Jul 29 03:14:21.304958 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.304942 2579 scope.go:117] "RemoveContainer" containerID="f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821" Jul 29 03:14:21.305232 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:21.305216 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821\": container with ID starting with f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821 not found: ID does not exist" containerID="f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821" Jul 29 03:14:21.305295 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.305243 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821"} err="failed to get container status \"f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821\": rpc error: code = NotFound desc = could not find container \"f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821\": container with ID starting with f10ec59b7c700efdc4d9ce3665891b2d4deb9e4cefe8208a30a1914308fa7821 not found: ID does not exist" Jul 29 03:14:21.345946 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.345846 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f297412-7c03-4c04-bc8d-b4bb81803ebc-openshift-service-ca\") pod \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " Jul 29 03:14:21.345946 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.345893 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls\") pod \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " Jul 29 03:14:21.346210 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.345951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thbgc\" (UniqueName: \"kubernetes.io/projected/2f297412-7c03-4c04-bc8d-b4bb81803ebc-kube-api-access-thbgc\") pod \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\" (UID: \"2f297412-7c03-4c04-bc8d-b4bb81803ebc\") " Jul 29 03:14:21.346277 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.346250 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f297412-7c03-4c04-bc8d-b4bb81803ebc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f297412-7c03-4c04-bc8d-b4bb81803ebc" (UID: "2f297412-7c03-4c04-bc8d-b4bb81803ebc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:14:21.346340 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.346326 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f297412-7c03-4c04-bc8d-b4bb81803ebc-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:21.348060 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.348030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f297412-7c03-4c04-bc8d-b4bb81803ebc" (UID: "2f297412-7c03-4c04-bc8d-b4bb81803ebc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:14:21.348186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.348065 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f297412-7c03-4c04-bc8d-b4bb81803ebc-kube-api-access-thbgc" (OuterVolumeSpecName: "kube-api-access-thbgc") pod "2f297412-7c03-4c04-bc8d-b4bb81803ebc" (UID: "2f297412-7c03-4c04-bc8d-b4bb81803ebc"). InnerVolumeSpecName "kube-api-access-thbgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:14:21.447192 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.447152 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f297412-7c03-4c04-bc8d-b4bb81803ebc-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:21.447192 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.447186 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thbgc\" (UniqueName: \"kubernetes.io/projected/2f297412-7c03-4c04-bc8d-b4bb81803ebc-kube-api-access-thbgc\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:21.617803 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.617767 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484"] Jul 29 03:14:21.622945 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:21.622918 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2f84e8-58bb9bf9b-hd484"] Jul 29 03:14:22.452759 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:22.452721 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f297412-7c03-4c04-bc8d-b4bb81803ebc" path="/var/lib/kubelet/pods/2f297412-7c03-4c04-bc8d-b4bb81803ebc/volumes" Jul 29 03:14:27.577882 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:27.577846 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf"] Jul 29 03:14:27.578338 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:27.578082 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" containerID="cri-o://b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9" gracePeriod=30 Jul 29 03:14:28.476253 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476214 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh"] Jul 29 03:14:28.476807 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476780 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" Jul 29 03:14:28.476807 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476804 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" Jul 29 03:14:28.476959 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476836 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecf796d-e626-44dd-aecb-6058d21db1fe" containerName="maas-api" Jul 29 03:14:28.476959 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476844 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecf796d-e626-44dd-aecb-6058d21db1fe" containerName="maas-api" Jul 29 03:14:28.476959 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476857 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f297412-7c03-4c04-bc8d-b4bb81803ebc" containerName="maas-api" Jul 29 03:14:28.476959 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476867 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f297412-7c03-4c04-bc8d-b4bb81803ebc" containerName="maas-api" Jul 29 03:14:28.477110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476968 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecf796d-e626-44dd-aecb-6058d21db1fe" containerName="maas-api" Jul 29 03:14:28.477110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476977 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f297412-7c03-4c04-bc8d-b4bb81803ebc" containerName="maas-api" Jul 29 03:14:28.477110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476989 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" Jul 29 03:14:28.477110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.476998 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" Jul 29 03:14:28.479740 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.479723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.481603 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.481582 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1d2f38-openshift-default-dockercfg-m2gd8\"" Jul 29 03:14:28.491243 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.491219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh"] Jul 29 03:14:28.511943 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.511911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-data\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512129 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.511958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-podinfo\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512129 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.512000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjb79\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-kube-api-access-zjb79\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512129 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.512034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-token\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512129 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.512064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-certs\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512296 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.512137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8273fb7b-29cf-4e32-8e27-ed5555afe011-istiod-ca-cert\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512296 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.512161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-credential-socket\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512296 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.512200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-envoy\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.512296 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.512238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-socket\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613009 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.612967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-envoy\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-socket\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-data\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-podinfo\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjb79\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-kube-api-access-zjb79\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-token\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-certs\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-socket\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613899 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-data\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613899 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613595 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-certs\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613899 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8273fb7b-29cf-4e32-8e27-ed5555afe011-istiod-ca-cert\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.613899 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.613781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-credential-socket\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.614181 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.614138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-credential-socket\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.614467 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.614444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8273fb7b-29cf-4e32-8e27-ed5555afe011-istiod-ca-cert\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.615355 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.615335 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-envoy\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.615793 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.615777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-podinfo\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.623411 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.623385 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjb79\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-kube-api-access-zjb79\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.624715 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.624693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-token\") pod \"e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.791159 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.791063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:28.933294 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.933266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh"] Jul 29 03:14:28.935600 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:14:28.935567 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8273fb7b_29cf_4e32_8e27_ed5555afe011.slice/crio-3e7c4565622c29dfee970c8eefecf5485df05ca1872ede098fa709680b439e72 WatchSource:0}: Error finding container 3e7c4565622c29dfee970c8eefecf5485df05ca1872ede098fa709680b439e72: Status 404 returned error can't find the container with id 3e7c4565622c29dfee970c8eefecf5485df05ca1872ede098fa709680b439e72 Jul 29 03:14:28.937720 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.937684 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:28.937793 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.937759 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:28.937793 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:28.937788 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:29.338232 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:29.338192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" event={"ID":"8273fb7b-29cf-4e32-8e27-ed5555afe011","Type":"ContainerStarted","Data":"93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430"} Jul 29 03:14:29.338429 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:29.338247 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" event={"ID":"8273fb7b-29cf-4e32-8e27-ed5555afe011","Type":"ContainerStarted","Data":"3e7c4565622c29dfee970c8eefecf5485df05ca1872ede098fa709680b439e72"} Jul 29 03:14:29.358339 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:29.358282 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" podStartSLOduration=1.358264401 podStartE2EDuration="1.358264401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:14:29.356327249 +0000 UTC m=+2359.416802644" watchObservedRunningTime="2026-07-29 03:14:29.358264401 +0000 UTC m=+2359.418739786" Jul 29 03:14:29.791638 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:29.791601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:29.796768 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:29.796743 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:30.343255 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:30.343223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:30.344131 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:30.344107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:31.449152 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.449112 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r"] Jul 29 03:14:31.449641 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.449623 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" Jul 29 03:14:31.449641 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.449641 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9030af-9e8c-4527-9cd9-8ce1865fe07f" containerName="istio-proxy" Jul 29 03:14:31.452277 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.452251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.454396 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.454368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1d2f38-serving-cert\"" Jul 29 03:14:31.462745 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.462709 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r"] Jul 29 03:14:31.540070 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.540010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.540291 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.540168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.540291 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.540240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdbs\" (UniqueName: \"kubernetes.io/projected/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-kube-api-access-kvdbs\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.641749 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.641711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.641945 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.641785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdbs\" (UniqueName: \"kubernetes.io/projected/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-kube-api-access-kvdbs\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.641945 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.641836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.642053 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:31.641973 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-serving-cert: secret "maas-api-e2e-api-a-1d2f38-serving-cert" not found Jul 29 03:14:31.642103 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:31.642068 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls podName:84a1bb42-7a29-47ea-8af3-be8a8a6330fc nodeName:}" failed. No retries permitted until 2026-07-29 03:14:32.142048113 +0000 UTC m=+2362.202523490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls") pod "maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" (UID: "84a1bb42-7a29-47ea-8af3-be8a8a6330fc") : secret "maas-api-e2e-api-a-1d2f38-serving-cert" not found Jul 29 03:14:31.642373 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.642353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:31.650548 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:31.650516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdbs\" (UniqueName: \"kubernetes.io/projected/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-kube-api-access-kvdbs\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:32.146912 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.146875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:32.149336 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.149314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls\") pod \"maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:32.318307 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.318262 2579 patch_prober.go:28] interesting pod/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 03:14:32.318482 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.318352 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:14:32.365593 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.365559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:32.514875 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.514851 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r"] Jul 29 03:14:32.516753 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:14:32.516722 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a1bb42_7a29_47ea_8af3_be8a8a6330fc.slice/crio-c6f557bd83734d75477e496bddc5ed94a7d8ea1dc6b7a80b31358ba3b5f5363f WatchSource:0}: Error finding container c6f557bd83734d75477e496bddc5ed94a7d8ea1dc6b7a80b31358ba3b5f5363f: Status 404 returned error can't find the container with id c6f557bd83734d75477e496bddc5ed94a7d8ea1dc6b7a80b31358ba3b5f5363f Jul 29 03:14:32.518186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.518167 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 03:14:32.834674 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.834646 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:14:32.854030 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.853900 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020422d8-17e1-4816-8cd7-2c18e9698235-istiod-ca-cert\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854030 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.853971 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020422d8-17e1-4816-8cd7-2c18e9698235-istio-podinfo\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854109 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-socket\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854173 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-credential-socket\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854197 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrj7q\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-kube-api-access-hrj7q\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854228 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-certs\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854251 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-envoy\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854533 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-istio-token\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854533 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854305 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-data\") pod \"020422d8-17e1-4816-8cd7-2c18e9698235\" (UID: \"020422d8-17e1-4816-8cd7-2c18e9698235\") " Jul 29 03:14:32.854533 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854371 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020422d8-17e1-4816-8cd7-2c18e9698235-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:14:32.854694 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854595 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020422d8-17e1-4816-8cd7-2c18e9698235-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.854748 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854705 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:32.854826 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.854797 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:32.855686 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.855521 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-data" (OuterVolumeSpecName: "istio-data") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:32.860561 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.858868 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:32.861533 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.861498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-istio-token" (OuterVolumeSpecName: "istio-token") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:14:32.862163 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.862133 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-kube-api-access-hrj7q" (OuterVolumeSpecName: "kube-api-access-hrj7q") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "kube-api-access-hrj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:14:32.862385 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.862344 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:14:32.862385 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.862349 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/020422d8-17e1-4816-8cd7-2c18e9698235-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "020422d8-17e1-4816-8cd7-2c18e9698235" (UID: "020422d8-17e1-4816-8cd7-2c18e9698235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:14:32.956270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956169 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.956270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956210 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.956270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956222 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020422d8-17e1-4816-8cd7-2c18e9698235-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.956270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956235 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.956270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956250 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.956270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956265 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrj7q\" (UniqueName: \"kubernetes.io/projected/020422d8-17e1-4816-8cd7-2c18e9698235-kube-api-access-hrj7q\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.956270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956278 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:32.956722 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:32.956289 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020422d8-17e1-4816-8cd7-2c18e9698235-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:14:33.363228 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.363138 2579 generic.go:356] "Generic (PLEG): container finished" podID="020422d8-17e1-4816-8cd7-2c18e9698235" containerID="b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9" exitCode=0 Jul 29 03:14:33.363228 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.363195 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" Jul 29 03:14:33.363228 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.363207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" event={"ID":"020422d8-17e1-4816-8cd7-2c18e9698235","Type":"ContainerDied","Data":"b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9"} Jul 29 03:14:33.363494 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.363264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf" event={"ID":"020422d8-17e1-4816-8cd7-2c18e9698235","Type":"ContainerDied","Data":"16eb532042c7be34d05b08e36d7c6344627228aa1768aed00375fd1fd972e8c6"} Jul 29 03:14:33.363494 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.363296 2579 scope.go:117] "RemoveContainer" containerID="b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9" Jul 29 03:14:33.365121 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.365090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" event={"ID":"84a1bb42-7a29-47ea-8af3-be8a8a6330fc","Type":"ContainerStarted","Data":"d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d"} Jul 29 03:14:33.365253 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.365139 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" event={"ID":"84a1bb42-7a29-47ea-8af3-be8a8a6330fc","Type":"ContainerStarted","Data":"c6f557bd83734d75477e496bddc5ed94a7d8ea1dc6b7a80b31358ba3b5f5363f"} Jul 29 03:14:33.365253 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.365248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:33.374160 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.373979 2579 scope.go:117] "RemoveContainer" containerID="ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f" Jul 29 03:14:33.384936 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.384881 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" podStartSLOduration=1.853502496 podStartE2EDuration="2.384860539s" totalImagesPullingTime="531.358043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:14:33.382052253 +0000 UTC m=+2363.442527668" watchObservedRunningTime="2026-07-29 03:14:33.384860539 +0000 UTC m=+2363.445335927" Jul 29 03:14:33.386787 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.386762 2579 scope.go:117] "RemoveContainer" containerID="b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9" Jul 29 03:14:33.387208 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:33.387151 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9\": container with ID starting with b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9 not found: ID does not exist" containerID="b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9" Jul 29 03:14:33.387572 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.387537 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9"} err="failed to get container status \"b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9\": rpc error: code = NotFound desc = could not find container \"b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9\": container with ID starting with b9a1bf333ae73b2b7871d08e11206a5aadfb0568ea4bf08ac3f38ac663c1fed9 not found: ID does not exist" Jul 29 03:14:33.387572 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.387570 2579 scope.go:117] "RemoveContainer" containerID="ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f" Jul 29 03:14:33.387901 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:14:33.387880 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f\": container with ID starting with ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f not found: ID does not exist" containerID="ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f" Jul 29 03:14:33.387973 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.387910 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f"} err="failed to get container status \"ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f\": rpc error: code = NotFound desc = could not find container \"ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f\": container with ID starting with ad067d8e48219b53fb5a9b6d1ec6dc6c7f914a789c69f11b1957331a53602c2f not found: ID does not exist" Jul 29 03:14:33.396530 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.396503 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf"] Jul 29 03:14:33.402765 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:33.402742 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2f84e8-openshift-default-844f5c4d46-jdggf"] Jul 29 03:14:34.453759 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:34.453726 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" path="/var/lib/kubelet/pods/020422d8-17e1-4816-8cd7-2c18e9698235/volumes" Jul 29 03:14:39.379374 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:39.379344 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:14:44.426786 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:44.426749 2579 generic.go:356] "Generic (PLEG): container finished" podID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerID="93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430" exitCode=0 Jul 29 03:14:44.427247 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:44.426821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" event={"ID":"8273fb7b-29cf-4e32-8e27-ed5555afe011","Type":"ContainerDied","Data":"93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430"} Jul 29 03:14:44.427317 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:44.427258 2579 scope.go:117] "RemoveContainer" containerID="93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430" Jul 29 03:14:44.428273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:44.428242 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:44.428383 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:44.428313 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:44.428383 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:44.428343 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:45.343890 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:45.343852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:45.433611 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:45.433573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" event={"ID":"8273fb7b-29cf-4e32-8e27-ed5555afe011","Type":"ContainerStarted","Data":"2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c"} Jul 29 03:14:45.791578 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:45.791537 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:45.792859 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:45.792828 2579 patch_prober.go:28] interesting pod/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 29 03:14:45.792992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:45.792881 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 29 03:14:46.791720 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:46.791682 2579 patch_prober.go:28] interesting pod/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 29 03:14:46.792170 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:46.791755 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 29 03:14:47.792223 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:47.792179 2579 patch_prober.go:28] interesting pod/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 29 03:14:47.792628 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:47.792259 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 29 03:14:48.791955 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:48.791914 2579 patch_prober.go:28] interesting pod/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 29 03:14:48.792167 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:48.791979 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 29 03:14:49.795750 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:49.795716 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:49.796260 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:49.796113 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:49.796893 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:49.796873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:14:57.031198 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.031161 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm"] Jul 29 03:14:57.031681 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.031548 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" Jul 29 03:14:57.031681 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.031559 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" Jul 29 03:14:57.031681 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.031644 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" Jul 29 03:14:57.033766 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.033738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.035761 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.035739 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bdd1b4-openshift-default-dockercfg-6kqhs\"" Jul 29 03:14:57.045495 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.045471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm"] Jul 29 03:14:57.188726 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.188687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26ns\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-kube-api-access-v26ns\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.188910 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.188751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-credential-socket\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.188910 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.188835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-data\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.188910 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.188874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-podinfo\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.188910 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.188906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-certs\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.189110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.188952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-token\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.189110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.188980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-socket\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.189110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.189085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istiod-ca-cert\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.189209 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.189111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-envoy\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290208 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istiod-ca-cert\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290208 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-envoy\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290208 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26ns\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-kube-api-access-v26ns\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290492 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-credential-socket\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290492 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-data\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290492 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-podinfo\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290492 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-certs\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290492 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-token\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290492 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-socket\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.290979 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.290951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istiod-ca-cert\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.291209 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.291007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-certs\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.291334 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.291052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-data\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.291334 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.291125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-socket\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.291334 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.291301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-credential-socket\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.292451 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.292430 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-envoy\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.292756 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.292736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-podinfo\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.297684 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.297662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26ns\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-kube-api-access-v26ns\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.297842 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.297825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-token\") pod \"e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.348027 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.347980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:57.493963 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.493937 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm"] Jul 29 03:14:57.496573 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:14:57.496532 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758759d9_cf5f_46d0_93a0_bbb9faf85a66.slice/crio-0a9ebdb272148dde7e9cccec37837cdc51b222e1c22e11bc3ff31368bee25e73 WatchSource:0}: Error finding container 0a9ebdb272148dde7e9cccec37837cdc51b222e1c22e11bc3ff31368bee25e73: Status 404 returned error can't find the container with id 0a9ebdb272148dde7e9cccec37837cdc51b222e1c22e11bc3ff31368bee25e73 Jul 29 03:14:57.498769 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.498731 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:57.498851 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.498801 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:57.498851 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:57.498829 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:14:58.498978 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:58.498938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" event={"ID":"758759d9-cf5f-46d0-93a0-bbb9faf85a66","Type":"ContainerStarted","Data":"31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1"} Jul 29 03:14:58.499397 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:58.498997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" event={"ID":"758759d9-cf5f-46d0-93a0-bbb9faf85a66","Type":"ContainerStarted","Data":"0a9ebdb272148dde7e9cccec37837cdc51b222e1c22e11bc3ff31368bee25e73"} Jul 29 03:14:58.521674 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:58.521621 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" podStartSLOduration=1.5216043240000001 podStartE2EDuration="1.521604324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:14:58.519251247 +0000 UTC m=+2388.579726633" watchObservedRunningTime="2026-07-29 03:14:58.521604324 +0000 UTC m=+2388.582079709" Jul 29 03:14:59.348849 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:59.348806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:59.354272 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:59.354243 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:59.503659 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:59.503630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:14:59.504893 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:14:59.504873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:15:00.538313 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.538277 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4"] Jul 29 03:15:00.539210 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.539009 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" Jul 29 03:15:00.539210 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.539061 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" Jul 29 03:15:00.539210 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.539186 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="020422d8-17e1-4816-8cd7-2c18e9698235" containerName="istio-proxy" Jul 29 03:15:00.541396 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.541376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.543304 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.543284 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bdd1b4-serving-cert\"" Jul 29 03:15:00.551078 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.550834 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4"] Jul 29 03:15:00.725619 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.725579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033c87cb-79fb-46b7-b858-8261cab30a9d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.725827 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.725647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033c87cb-79fb-46b7-b858-8261cab30a9d-maas-api-tls\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.725827 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.725738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkcq5\" (UniqueName: \"kubernetes.io/projected/033c87cb-79fb-46b7-b858-8261cab30a9d-kube-api-access-kkcq5\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.826692 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.826585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033c87cb-79fb-46b7-b858-8261cab30a9d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.826692 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.826651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033c87cb-79fb-46b7-b858-8261cab30a9d-maas-api-tls\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.826692 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.826678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkcq5\" (UniqueName: \"kubernetes.io/projected/033c87cb-79fb-46b7-b858-8261cab30a9d-kube-api-access-kkcq5\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.827370 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.827341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033c87cb-79fb-46b7-b858-8261cab30a9d-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.829069 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.829048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033c87cb-79fb-46b7-b858-8261cab30a9d-maas-api-tls\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.833978 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.833957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkcq5\" (UniqueName: \"kubernetes.io/projected/033c87cb-79fb-46b7-b858-8261cab30a9d-kube-api-access-kkcq5\") pod \"maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.856144 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.856115 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:00.999474 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:00.999435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4"] Jul 29 03:15:01.002286 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:15:01.002260 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033c87cb_79fb_46b7_b858_8261cab30a9d.slice/crio-9d75729bf9cdeef491111285c7521140eeb4512772e27df539806b7e8b881d15 WatchSource:0}: Error finding container 9d75729bf9cdeef491111285c7521140eeb4512772e27df539806b7e8b881d15: Status 404 returned error can't find the container with id 9d75729bf9cdeef491111285c7521140eeb4512772e27df539806b7e8b881d15 Jul 29 03:15:01.514715 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:01.514673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" event={"ID":"033c87cb-79fb-46b7-b858-8261cab30a9d","Type":"ContainerStarted","Data":"9d75729bf9cdeef491111285c7521140eeb4512772e27df539806b7e8b881d15"} Jul 29 03:15:02.521322 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:02.521285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" event={"ID":"033c87cb-79fb-46b7-b858-8261cab30a9d","Type":"ContainerStarted","Data":"e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a"} Jul 29 03:15:02.521701 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:02.521346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:02.540251 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:02.540199 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" podStartSLOduration=1.998594349 podStartE2EDuration="2.540178838s" totalImagesPullingTime="541.584489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:15:02.537655546 +0000 UTC m=+2392.598130932" watchObservedRunningTime="2026-07-29 03:15:02.540178838 +0000 UTC m=+2392.600654222" Jul 29 03:15:08.531815 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:08.531782 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:15:10.505214 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:10.505179 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:15:10.507990 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:10.507963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:15:13.578085 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:13.578053 2579 generic.go:356] "Generic (PLEG): container finished" podID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerID="31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1" exitCode=0 Jul 29 03:15:13.578532 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:13.578127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" event={"ID":"758759d9-cf5f-46d0-93a0-bbb9faf85a66","Type":"ContainerDied","Data":"31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1"} Jul 29 03:15:13.578580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:13.578530 2579 scope.go:117] "RemoveContainer" containerID="31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1" Jul 29 03:15:13.579622 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:13.579591 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:15:13.579744 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:13.579678 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:15:13.579744 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:13.579726 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:15:14.504008 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:14.503964 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:15:14.585892 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:14.585853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" event={"ID":"758759d9-cf5f-46d0-93a0-bbb9faf85a66","Type":"ContainerStarted","Data":"9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03"} Jul 29 03:15:15.348221 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:15.348172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:15:15.349591 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:15.349563 2579 patch_prober.go:28] interesting pod/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 29 03:15:15.349737 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:15.349616 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 29 03:15:16.348814 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:16.348770 2579 patch_prober.go:28] interesting pod/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 29 03:15:16.349331 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:16.348839 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 29 03:15:17.348738 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:17.348703 2579 patch_prober.go:28] interesting pod/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 29 03:15:17.348919 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:17.348765 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 29 03:15:18.352483 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:18.352453 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:15:18.352849 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:18.352775 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:15:18.353510 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:18.353494 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:15:43.936524 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:43.936488 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r"] Jul 29 03:15:43.937005 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:43.936701 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" podUID="84a1bb42-7a29-47ea-8af3-be8a8a6330fc" containerName="maas-api" containerID="cri-o://d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d" gracePeriod=30 Jul 29 03:15:44.217316 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.217286 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:15:44.257539 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.257496 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvdbs\" (UniqueName: \"kubernetes.io/projected/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-kube-api-access-kvdbs\") pod \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " Jul 29 03:15:44.257736 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.257556 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls\") pod \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " Jul 29 03:15:44.257736 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.257601 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-openshift-service-ca\") pod \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\" (UID: \"84a1bb42-7a29-47ea-8af3-be8a8a6330fc\") " Jul 29 03:15:44.258280 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.258249 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84a1bb42-7a29-47ea-8af3-be8a8a6330fc" (UID: "84a1bb42-7a29-47ea-8af3-be8a8a6330fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:15:44.259868 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.259836 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84a1bb42-7a29-47ea-8af3-be8a8a6330fc" (UID: "84a1bb42-7a29-47ea-8af3-be8a8a6330fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:15:44.259989 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.259966 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-kube-api-access-kvdbs" (OuterVolumeSpecName: "kube-api-access-kvdbs") pod "84a1bb42-7a29-47ea-8af3-be8a8a6330fc" (UID: "84a1bb42-7a29-47ea-8af3-be8a8a6330fc"). InnerVolumeSpecName "kube-api-access-kvdbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:15:44.359080 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.359044 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdbs\" (UniqueName: \"kubernetes.io/projected/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-kube-api-access-kvdbs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:44.359080 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.359078 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:44.359080 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.359087 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a1bb42-7a29-47ea-8af3-be8a8a6330fc-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:44.734003 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.733910 2579 generic.go:356] "Generic (PLEG): container finished" podID="84a1bb42-7a29-47ea-8af3-be8a8a6330fc" containerID="d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d" exitCode=0 Jul 29 03:15:44.734003 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.733955 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" Jul 29 03:15:44.734229 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.733997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" event={"ID":"84a1bb42-7a29-47ea-8af3-be8a8a6330fc","Type":"ContainerDied","Data":"d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d"} Jul 29 03:15:44.734229 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.734065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r" event={"ID":"84a1bb42-7a29-47ea-8af3-be8a8a6330fc","Type":"ContainerDied","Data":"c6f557bd83734d75477e496bddc5ed94a7d8ea1dc6b7a80b31358ba3b5f5363f"} Jul 29 03:15:44.734229 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.734088 2579 scope.go:117] "RemoveContainer" containerID="d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d" Jul 29 03:15:44.742469 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.742450 2579 scope.go:117] "RemoveContainer" containerID="d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d" Jul 29 03:15:44.742749 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:15:44.742728 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d\": container with ID starting with d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d not found: ID does not exist" containerID="d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d" Jul 29 03:15:44.742791 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.742757 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d"} err="failed to get container status \"d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d\": rpc error: code = NotFound desc = could not find container \"d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d\": container with ID starting with d4b1413ae866fca1abbfd819a68bb845286b5643656a025ccb43497fd3506e6d not found: ID does not exist" Jul 29 03:15:44.748489 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.748466 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r"] Jul 29 03:15:44.755139 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:44.755112 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1d2f38-66447bf97c-zhl5r"] Jul 29 03:15:46.452886 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:46.452841 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a1bb42-7a29-47ea-8af3-be8a8a6330fc" path="/var/lib/kubelet/pods/84a1bb42-7a29-47ea-8af3-be8a8a6330fc/volumes" Jul 29 03:15:50.438314 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:50.438276 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh"] Jul 29 03:15:50.438823 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:50.438550 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" containerID="cri-o://2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c" gracePeriod=30 Jul 29 03:15:55.706213 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.706149 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:15:55.769350 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769320 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjb79\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-kube-api-access-zjb79\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769382 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-socket\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769417 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-token\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-certs\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769476 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-podinfo\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769518 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-credential-socket\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-data\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769863 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769587 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-envoy\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769863 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769613 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8273fb7b-29cf-4e32-8e27-ed5555afe011-istiod-ca-cert\") pod \"8273fb7b-29cf-4e32-8e27-ed5555afe011\" (UID: \"8273fb7b-29cf-4e32-8e27-ed5555afe011\") " Jul 29 03:15:55.769863 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:15:55.769863 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.769769 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:15:55.770077 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.770033 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.770077 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.770057 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.770192 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.770152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:15:55.770254 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.770184 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8273fb7b-29cf-4e32-8e27-ed5555afe011-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:15:55.770830 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.770803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-data" (OuterVolumeSpecName: "istio-data") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:15:55.771615 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.771593 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-token" (OuterVolumeSpecName: "istio-token") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:15:55.771729 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.771679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-kube-api-access-zjb79" (OuterVolumeSpecName: "kube-api-access-zjb79") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "kube-api-access-zjb79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:15:55.772002 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.771983 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:15:55.772100 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.771988 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8273fb7b-29cf-4e32-8e27-ed5555afe011" (UID: "8273fb7b-29cf-4e32-8e27-ed5555afe011"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:15:55.795571 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.795538 2579 generic.go:356] "Generic (PLEG): container finished" podID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerID="2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c" exitCode=0 Jul 29 03:15:55.795717 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.795588 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" Jul 29 03:15:55.795717 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.795602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" event={"ID":"8273fb7b-29cf-4e32-8e27-ed5555afe011","Type":"ContainerDied","Data":"2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c"} Jul 29 03:15:55.795717 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.795657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh" event={"ID":"8273fb7b-29cf-4e32-8e27-ed5555afe011","Type":"ContainerDied","Data":"3e7c4565622c29dfee970c8eefecf5485df05ca1872ede098fa709680b439e72"} Jul 29 03:15:55.795717 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.795686 2579 scope.go:117] "RemoveContainer" containerID="2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c" Jul 29 03:15:55.805259 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.805232 2579 scope.go:117] "RemoveContainer" containerID="93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430" Jul 29 03:15:55.813053 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.812999 2579 scope.go:117] "RemoveContainer" containerID="2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c" Jul 29 03:15:55.813305 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:15:55.813286 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c\": container with ID starting with 2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c not found: ID does not exist" containerID="2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c" Jul 29 03:15:55.813364 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.813314 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c"} err="failed to get container status \"2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c\": rpc error: code = NotFound desc = could not find container \"2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c\": container with ID starting with 2208aee65d03f3af3552f479c66c628b2f0b856c8746b4616a04ed30be46869c not found: ID does not exist" Jul 29 03:15:55.813364 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.813334 2579 scope.go:117] "RemoveContainer" containerID="93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430" Jul 29 03:15:55.813584 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:15:55.813569 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430\": container with ID starting with 93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430 not found: ID does not exist" containerID="93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430" Jul 29 03:15:55.813621 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.813592 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430"} err="failed to get container status \"93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430\": rpc error: code = NotFound desc = could not find container \"93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430\": container with ID starting with 93b9080e6cdbe6ddca36fb83673d66a802a105670e798198510dd07d4fe73430 not found: ID does not exist" Jul 29 03:15:55.816718 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.816696 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh"] Jul 29 03:15:55.820220 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.820199 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1d2f38-openshift-default-6db97fb9b-xwhsh"] Jul 29 03:15:55.870634 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.870605 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.870634 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.870630 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.870634 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.870642 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.870858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.870649 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8273fb7b-29cf-4e32-8e27-ed5555afe011-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.870858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.870657 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjb79\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-kube-api-access-zjb79\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.870858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.870666 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:55.870858 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:55.870673 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8273fb7b-29cf-4e32-8e27-ed5555afe011-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:15:56.453135 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:15:56.453100 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" path="/var/lib/kubelet/pods/8273fb7b-29cf-4e32-8e27-ed5555afe011/volumes" Jul 29 03:16:06.341088 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.341052 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4"] Jul 29 03:16:06.341659 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.341279 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" podUID="033c87cb-79fb-46b7-b858-8261cab30a9d" containerName="maas-api" containerID="cri-o://e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a" gracePeriod=30 Jul 29 03:16:06.595594 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.595523 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:16:06.670954 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.670923 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033c87cb-79fb-46b7-b858-8261cab30a9d-openshift-service-ca\") pod \"033c87cb-79fb-46b7-b858-8261cab30a9d\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " Jul 29 03:16:06.671176 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.671111 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033c87cb-79fb-46b7-b858-8261cab30a9d-maas-api-tls\") pod \"033c87cb-79fb-46b7-b858-8261cab30a9d\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " Jul 29 03:16:06.671176 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.671168 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkcq5\" (UniqueName: \"kubernetes.io/projected/033c87cb-79fb-46b7-b858-8261cab30a9d-kube-api-access-kkcq5\") pod \"033c87cb-79fb-46b7-b858-8261cab30a9d\" (UID: \"033c87cb-79fb-46b7-b858-8261cab30a9d\") " Jul 29 03:16:06.671330 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.671302 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033c87cb-79fb-46b7-b858-8261cab30a9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "033c87cb-79fb-46b7-b858-8261cab30a9d" (UID: "033c87cb-79fb-46b7-b858-8261cab30a9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:16:06.671474 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.671458 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033c87cb-79fb-46b7-b858-8261cab30a9d-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:06.673146 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.673126 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033c87cb-79fb-46b7-b858-8261cab30a9d-kube-api-access-kkcq5" (OuterVolumeSpecName: "kube-api-access-kkcq5") pod "033c87cb-79fb-46b7-b858-8261cab30a9d" (UID: "033c87cb-79fb-46b7-b858-8261cab30a9d"). InnerVolumeSpecName "kube-api-access-kkcq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:16:06.673281 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.673263 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033c87cb-79fb-46b7-b858-8261cab30a9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "033c87cb-79fb-46b7-b858-8261cab30a9d" (UID: "033c87cb-79fb-46b7-b858-8261cab30a9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:16:06.772059 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.772002 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033c87cb-79fb-46b7-b858-8261cab30a9d-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:06.772059 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.772054 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkcq5\" (UniqueName: \"kubernetes.io/projected/033c87cb-79fb-46b7-b858-8261cab30a9d-kube-api-access-kkcq5\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:06.849838 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.849747 2579 generic.go:356] "Generic (PLEG): container finished" podID="033c87cb-79fb-46b7-b858-8261cab30a9d" containerID="e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a" exitCode=0 Jul 29 03:16:06.849838 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.849792 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" Jul 29 03:16:06.849838 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.849800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" event={"ID":"033c87cb-79fb-46b7-b858-8261cab30a9d","Type":"ContainerDied","Data":"e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a"} Jul 29 03:16:06.850128 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.849842 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4" event={"ID":"033c87cb-79fb-46b7-b858-8261cab30a9d","Type":"ContainerDied","Data":"9d75729bf9cdeef491111285c7521140eeb4512772e27df539806b7e8b881d15"} Jul 29 03:16:06.850128 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.849862 2579 scope.go:117] "RemoveContainer" containerID="e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a" Jul 29 03:16:06.860248 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.860219 2579 scope.go:117] "RemoveContainer" containerID="e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a" Jul 29 03:16:06.860523 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:16:06.860505 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a\": container with ID starting with e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a not found: ID does not exist" containerID="e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a" Jul 29 03:16:06.860575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.860533 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a"} err="failed to get container status \"e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a\": rpc error: code = NotFound desc = could not find container \"e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a\": container with ID starting with e1f98abbb995b25d7a8ff64306d42335e5cefdb2bd2a390c14b9b090b2467d1a not found: ID does not exist" Jul 29 03:16:06.871828 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.871798 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4"] Jul 29 03:16:06.875236 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:06.875213 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdd1b4-65fcbfb875-zb7f4"] Jul 29 03:16:08.453197 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:08.453163 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="033c87cb-79fb-46b7-b858-8261cab30a9d" path="/var/lib/kubelet/pods/033c87cb-79fb-46b7-b858-8261cab30a9d/volumes" Jul 29 03:16:12.853417 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:12.853382 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm"] Jul 29 03:16:12.853890 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:12.853648 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" containerID="cri-o://9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03" gracePeriod=30 Jul 29 03:16:13.844597 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.844552 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8"] Jul 29 03:16:13.845168 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845149 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" Jul 29 03:16:13.845274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845170 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" Jul 29 03:16:13.845274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845186 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a1bb42-7a29-47ea-8af3-be8a8a6330fc" containerName="maas-api" Jul 29 03:16:13.845274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845194 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a1bb42-7a29-47ea-8af3-be8a8a6330fc" containerName="maas-api" Jul 29 03:16:13.845274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845212 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033c87cb-79fb-46b7-b858-8261cab30a9d" containerName="maas-api" Jul 29 03:16:13.845274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845220 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="033c87cb-79fb-46b7-b858-8261cab30a9d" containerName="maas-api" Jul 29 03:16:13.845274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845233 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" Jul 29 03:16:13.845274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845241 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" Jul 29 03:16:13.845497 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845325 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a1bb42-7a29-47ea-8af3-be8a8a6330fc" containerName="maas-api" Jul 29 03:16:13.845497 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845337 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" Jul 29 03:16:13.845497 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845346 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8273fb7b-29cf-4e32-8e27-ed5555afe011" containerName="istio-proxy" Jul 29 03:16:13.845497 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.845353 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="033c87cb-79fb-46b7-b858-8261cab30a9d" containerName="maas-api" Jul 29 03:16:13.849389 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.849370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.852711 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.852659 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-f4870d-openshift-default-dockercfg-lz6nd\"" Jul 29 03:16:13.864526 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.864500 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8"] Jul 29 03:16:13.938122 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-token\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938122 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-socket\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938428 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-envoy\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938428 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938293 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-podinfo\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938428 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41093fff-8613-4fa9-87ad-10e0f6b194e5-istiod-ca-cert\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938428 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-data\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938428 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw86f\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-kube-api-access-hw86f\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938428 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-certs\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:13.938644 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:13.938476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-credential-socket\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039353 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-token\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039538 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-socket\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039538 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-envoy\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039666 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-podinfo\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039721 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41093fff-8613-4fa9-87ad-10e0f6b194e5-istiod-ca-cert\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039774 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-data\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039774 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw86f\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-kube-api-access-hw86f\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039884 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-certs\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039884 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-credential-socket\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.039985 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.039915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-socket\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.040387 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.040133 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-data\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.040387 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.040203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-credential-socket\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.040387 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.040343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-certs\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.040743 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.040723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41093fff-8613-4fa9-87ad-10e0f6b194e5-istiod-ca-cert\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.041640 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.041619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-envoy\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.042687 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.042665 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-podinfo\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.047913 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.047889 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-token\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.048063 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.048046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw86f\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-kube-api-access-hw86f\") pod \"e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.161671 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.161576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:14.302799 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.302772 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8"] Jul 29 03:16:14.304997 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:16:14.304967 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41093fff_8613_4fa9_87ad_10e0f6b194e5.slice/crio-b83a5ff2dace618a566f69243136de11e0e40aff0bb28e86ddaf5a07b4945873 WatchSource:0}: Error finding container b83a5ff2dace618a566f69243136de11e0e40aff0bb28e86ddaf5a07b4945873: Status 404 returned error can't find the container with id b83a5ff2dace618a566f69243136de11e0e40aff0bb28e86ddaf5a07b4945873 Jul 29 03:16:14.307530 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.307501 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:14.307601 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.307589 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:14.307647 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.307618 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:14.893414 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.893377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" event={"ID":"41093fff-8613-4fa9-87ad-10e0f6b194e5","Type":"ContainerStarted","Data":"c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a"} Jul 29 03:16:14.893839 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.893440 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" event={"ID":"41093fff-8613-4fa9-87ad-10e0f6b194e5","Type":"ContainerStarted","Data":"b83a5ff2dace618a566f69243136de11e0e40aff0bb28e86ddaf5a07b4945873"} Jul 29 03:16:14.912876 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:14.912824 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podStartSLOduration=1.9128053299999999 podStartE2EDuration="1.91280533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:16:14.910308246 +0000 UTC m=+2464.970783626" watchObservedRunningTime="2026-07-29 03:16:14.91280533 +0000 UTC m=+2464.973280715" Jul 29 03:16:15.162450 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:15.162353 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:15.167455 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:15.167425 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:15.898960 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:15.898933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:15.899976 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:15.899955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:16.933687 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:16.933645 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9"] Jul 29 03:16:16.937608 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:16.937585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:16.939810 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:16.939788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f4870d-serving-cert\"" Jul 29 03:16:16.948120 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:16.948090 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9"] Jul 29 03:16:17.068866 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.068827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b53d6c4-9fb2-43b6-996d-e5195c418824-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.068866 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.068865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lltx\" (UniqueName: \"kubernetes.io/projected/8b53d6c4-9fb2-43b6-996d-e5195c418824-kube-api-access-8lltx\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.069148 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.068903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.170653 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.170612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b53d6c4-9fb2-43b6-996d-e5195c418824-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.170653 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.170659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lltx\" (UniqueName: \"kubernetes.io/projected/8b53d6c4-9fb2-43b6-996d-e5195c418824-kube-api-access-8lltx\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.170908 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.170794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.170976 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:16:17.170940 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-serving-cert: secret "maas-api-e2e-shared-a-f4870d-serving-cert" not found Jul 29 03:16:17.171048 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:16:17.171041 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls podName:8b53d6c4-9fb2-43b6-996d-e5195c418824 nodeName:}" failed. No retries permitted until 2026-07-29 03:16:17.670997942 +0000 UTC m=+2467.731473316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls") pod "maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" (UID: "8b53d6c4-9fb2-43b6-996d-e5195c418824") : secret "maas-api-e2e-shared-a-f4870d-serving-cert" not found Jul 29 03:16:17.171543 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.171520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b53d6c4-9fb2-43b6-996d-e5195c418824-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.181626 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.181599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lltx\" (UniqueName: \"kubernetes.io/projected/8b53d6c4-9fb2-43b6-996d-e5195c418824-kube-api-access-8lltx\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.677381 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.677351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.679646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.679626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls\") pod \"maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.849989 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.849947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:17.994619 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:17.994526 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9"] Jul 29 03:16:17.996877 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:16:17.996828 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b53d6c4_9fb2_43b6_996d_e5195c418824.slice/crio-82653cf2f76f668cd2af67de12ed62db9210e1b96dfe8a768162e41605df16d3 WatchSource:0}: Error finding container 82653cf2f76f668cd2af67de12ed62db9210e1b96dfe8a768162e41605df16d3: Status 404 returned error can't find the container with id 82653cf2f76f668cd2af67de12ed62db9210e1b96dfe8a768162e41605df16d3 Jul 29 03:16:18.107795 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.107768 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:16:18.285186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285072 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v26ns\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-kube-api-access-v26ns\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285148 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-socket\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285178 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-certs\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285247 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-data\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285268 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-podinfo\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285300 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-credential-socket\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285325 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-envoy\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285365 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istiod-ca-cert\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285390 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-token\") pod \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\" (UID: \"758759d9-cf5f-46d0-93a0-bbb9faf85a66\") " Jul 29 03:16:18.285575 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285508 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:16:18.285920 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285645 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:16:18.285920 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285833 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:16:18.286056 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285969 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.286056 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.285990 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.286366 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.286317 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-data" (OuterVolumeSpecName: "istio-data") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:16:18.286502 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.286421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:16:18.287722 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.287691 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-token" (OuterVolumeSpecName: "istio-token") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:16:18.287838 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.287791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:16:18.287903 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.287883 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-kube-api-access-v26ns" (OuterVolumeSpecName: "kube-api-access-v26ns") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "kube-api-access-v26ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:16:18.288101 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.288082 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "758759d9-cf5f-46d0-93a0-bbb9faf85a66" (UID: "758759d9-cf5f-46d0-93a0-bbb9faf85a66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:16:18.386509 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.386475 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.386509 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.386507 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.386509 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.386516 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26ns\" (UniqueName: \"kubernetes.io/projected/758759d9-cf5f-46d0-93a0-bbb9faf85a66-kube-api-access-v26ns\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.386755 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.386526 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.386755 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.386534 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.386755 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.386542 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.386755 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.386550 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758759d9-cf5f-46d0-93a0-bbb9faf85a66-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:16:18.915067 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.914956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" event={"ID":"8b53d6c4-9fb2-43b6-996d-e5195c418824","Type":"ContainerStarted","Data":"dd8118e0ceee1fcd293171adadc5b4785fce2187f841a636c0f22ba84445e047"} Jul 29 03:16:18.915067 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.915008 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" event={"ID":"8b53d6c4-9fb2-43b6-996d-e5195c418824","Type":"ContainerStarted","Data":"82653cf2f76f668cd2af67de12ed62db9210e1b96dfe8a768162e41605df16d3"} Jul 29 03:16:18.915321 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.915139 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:18.916687 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.916661 2579 generic.go:356] "Generic (PLEG): container finished" podID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerID="9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03" exitCode=0 Jul 29 03:16:18.916818 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.916709 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" Jul 29 03:16:18.916818 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.916743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" event={"ID":"758759d9-cf5f-46d0-93a0-bbb9faf85a66","Type":"ContainerDied","Data":"9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03"} Jul 29 03:16:18.916818 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.916788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm" event={"ID":"758759d9-cf5f-46d0-93a0-bbb9faf85a66","Type":"ContainerDied","Data":"0a9ebdb272148dde7e9cccec37837cdc51b222e1c22e11bc3ff31368bee25e73"} Jul 29 03:16:18.916968 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.916820 2579 scope.go:117] "RemoveContainer" containerID="9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03" Jul 29 03:16:18.928892 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.928869 2579 scope.go:117] "RemoveContainer" containerID="31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1" Jul 29 03:16:18.931760 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.931627 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" podStartSLOduration=2.347015435 podStartE2EDuration="2.931607692s" totalImagesPullingTime="584.592257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:16:18.931111997 +0000 UTC m=+2468.991587383" watchObservedRunningTime="2026-07-29 03:16:18.931607692 +0000 UTC m=+2468.992083077" Jul 29 03:16:18.939706 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.939470 2579 scope.go:117] "RemoveContainer" containerID="9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03" Jul 29 03:16:18.940361 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:16:18.940333 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03\": container with ID starting with 9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03 not found: ID does not exist" containerID="9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03" Jul 29 03:16:18.940469 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.940369 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03"} err="failed to get container status \"9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03\": rpc error: code = NotFound desc = could not find container \"9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03\": container with ID starting with 9bf9d86b8f0babfd3bfe1df4ff33b55cc928ee3152ab77747a99839d9ae6ae03 not found: ID does not exist" Jul 29 03:16:18.940469 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.940393 2579 scope.go:117] "RemoveContainer" containerID="31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1" Jul 29 03:16:18.940693 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:16:18.940673 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1\": container with ID starting with 31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1 not found: ID does not exist" containerID="31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1" Jul 29 03:16:18.940746 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.940700 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1"} err="failed to get container status \"31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1\": rpc error: code = NotFound desc = could not find container \"31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1\": container with ID starting with 31a80dd0e82449fc9b7c2dfb5468246345b4ee52ee6835a6d71a0938ac052bf1 not found: ID does not exist" Jul 29 03:16:18.946238 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.946215 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm"] Jul 29 03:16:18.954559 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:18.954532 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bdd1b4-openshift-default-5485b8cf64-qpmmm"] Jul 29 03:16:20.452210 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:20.452177 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" path="/var/lib/kubelet/pods/758759d9-cf5f-46d0-93a0-bbb9faf85a66/volumes" Jul 29 03:16:24.928157 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:24.928128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:16:30.900946 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.900905 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 29 03:16:30.901409 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.900962 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 29 03:16:30.978700 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.978662 2579 generic.go:356] "Generic (PLEG): container finished" podID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerID="c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a" exitCode=0 Jul 29 03:16:30.978869 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.978735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" event={"ID":"41093fff-8613-4fa9-87ad-10e0f6b194e5","Type":"ContainerDied","Data":"c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a"} Jul 29 03:16:30.979148 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.979135 2579 scope.go:117] "RemoveContainer" containerID="c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a" Jul 29 03:16:30.980137 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.980109 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:30.980249 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.980202 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:30.980312 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:30.980259 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:31.985338 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:31.985308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" event={"ID":"41093fff-8613-4fa9-87ad-10e0f6b194e5","Type":"ContainerStarted","Data":"c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8"} Jul 29 03:16:32.162890 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:32.162843 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:32.163349 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:32.163315 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 29 03:16:32.163434 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:32.163386 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 29 03:16:33.162904 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:33.162851 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 29 03:16:33.163451 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:33.162929 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 29 03:16:34.162302 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:34.162261 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 29 03:16:34.162653 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:34.162613 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 29 03:16:35.162879 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:35.162837 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 29 03:16:35.163508 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:35.163478 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 29 03:16:36.166200 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:36.166167 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:36.166582 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:36.166512 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:36.167278 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:36.167263 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:16:42.508516 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.508470 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh"] Jul 29 03:16:42.509316 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.509292 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" Jul 29 03:16:42.509406 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.509324 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" Jul 29 03:16:42.509406 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.509352 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" Jul 29 03:16:42.509406 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.509366 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" Jul 29 03:16:42.509558 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.509537 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" Jul 29 03:16:42.509558 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.509551 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="758759d9-cf5f-46d0-93a0-bbb9faf85a66" containerName="istio-proxy" Jul 29 03:16:42.512459 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.512435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.514646 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.514623 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-19184d-openshift-default-dockercfg-jkhmb\"" Jul 29 03:16:42.521531 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.521079 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh"] Jul 29 03:16:42.616247 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-credential-socket\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616456 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-podinfo\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616456 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-token\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616456 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-certs\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616456 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ec4395-126a-4c03-b30e-256cf34bcae3-istiod-ca-cert\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616695 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-data\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616695 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-envoy\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616695 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-socket\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.616695 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.616612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85p5\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-kube-api-access-c85p5\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717343 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-podinfo\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-token\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-certs\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ec4395-126a-4c03-b30e-256cf34bcae3-istiod-ca-cert\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-data\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-envoy\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-socket\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717554 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85p5\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-kube-api-access-c85p5\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717932 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-credential-socket\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.717932 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-certs\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.718049 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.717954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-data\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.718049 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.718035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-credential-socket\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.718133 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.718056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-socket\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.718642 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.718619 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ec4395-126a-4c03-b30e-256cf34bcae3-istiod-ca-cert\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.720039 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.719998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-envoy\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.720273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.720256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-podinfo\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.724709 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.724682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-token\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.724795 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.724753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85p5\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-kube-api-access-c85p5\") pod \"e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.826300 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.826191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:42.968103 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.967735 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh"] Jul 29 03:16:42.970163 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:16:42.970137 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ec4395_126a_4c03_b30e_256cf34bcae3.slice/crio-cfec96fbe04936113a706ca54aebd436850bb9db6f43081e2ddc97ac32edd6a8 WatchSource:0}: Error finding container cfec96fbe04936113a706ca54aebd436850bb9db6f43081e2ddc97ac32edd6a8: Status 404 returned error can't find the container with id cfec96fbe04936113a706ca54aebd436850bb9db6f43081e2ddc97ac32edd6a8 Jul 29 03:16:42.972593 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.972562 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:42.972684 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.972630 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:42.972684 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:42.972663 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:43.044996 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:43.044963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" event={"ID":"77ec4395-126a-4c03-b30e-256cf34bcae3","Type":"ContainerStarted","Data":"c81c71bc062f9635d785f2f31d470d832e10c00f0c860857ec94db440d96e5f0"} Jul 29 03:16:43.045205 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:43.045048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" event={"ID":"77ec4395-126a-4c03-b30e-256cf34bcae3","Type":"ContainerStarted","Data":"cfec96fbe04936113a706ca54aebd436850bb9db6f43081e2ddc97ac32edd6a8"} Jul 29 03:16:43.068894 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:43.068839 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" podStartSLOduration=1.068821091 podStartE2EDuration="1.068821091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:16:43.067331224 +0000 UTC m=+2493.127806631" watchObservedRunningTime="2026-07-29 03:16:43.068821091 +0000 UTC m=+2493.129296478" Jul 29 03:16:43.827232 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:43.827182 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:44.832239 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:44.832203 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:45.057717 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.057676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:45.058618 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.058596 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:45.223935 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.223904 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7"] Jul 29 03:16:45.226394 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.226375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.228213 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.228194 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-19184d-serving-cert\"" Jul 29 03:16:45.239318 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.239295 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7"] Jul 29 03:16:45.343081 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.343044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrcz\" (UniqueName: \"kubernetes.io/projected/50a347c3-94ee-4cd4-9bc0-90c292316de8-kube-api-access-csrcz\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.343244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.343160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a347c3-94ee-4cd4-9bc0-90c292316de8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.343244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.343226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a347c3-94ee-4cd4-9bc0-90c292316de8-maas-api-tls\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.443949 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.443910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a347c3-94ee-4cd4-9bc0-90c292316de8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.444180 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.443978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a347c3-94ee-4cd4-9bc0-90c292316de8-maas-api-tls\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.444180 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.444028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrcz\" (UniqueName: \"kubernetes.io/projected/50a347c3-94ee-4cd4-9bc0-90c292316de8-kube-api-access-csrcz\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.444658 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.444628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a347c3-94ee-4cd4-9bc0-90c292316de8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.446504 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.446483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a347c3-94ee-4cd4-9bc0-90c292316de8-maas-api-tls\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.451876 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.451853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrcz\" (UniqueName: \"kubernetes.io/projected/50a347c3-94ee-4cd4-9bc0-90c292316de8-kube-api-access-csrcz\") pod \"maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.542736 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.542646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:45.687462 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:45.687436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7"] Jul 29 03:16:45.690144 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:16:45.690119 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a347c3_94ee_4cd4_9bc0_90c292316de8.slice/crio-41c963aecbe4171424822115bdb3f3b6a0176c8038c8ab7b9a49b2c0ae71b948 WatchSource:0}: Error finding container 41c963aecbe4171424822115bdb3f3b6a0176c8038c8ab7b9a49b2c0ae71b948: Status 404 returned error can't find the container with id 41c963aecbe4171424822115bdb3f3b6a0176c8038c8ab7b9a49b2c0ae71b948 Jul 29 03:16:46.075068 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:46.075004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" event={"ID":"50a347c3-94ee-4cd4-9bc0-90c292316de8","Type":"ContainerStarted","Data":"41c963aecbe4171424822115bdb3f3b6a0176c8038c8ab7b9a49b2c0ae71b948"} Jul 29 03:16:47.081814 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:47.081778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" event={"ID":"50a347c3-94ee-4cd4-9bc0-90c292316de8","Type":"ContainerStarted","Data":"0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280"} Jul 29 03:16:47.082296 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:47.081904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:47.100903 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:47.100843 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" podStartSLOduration=1.731651787 podStartE2EDuration="2.100826313s" totalImagesPullingTime="369.174526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:16:47.098389043 +0000 UTC m=+2497.158864467" watchObservedRunningTime="2026-07-29 03:16:47.100826313 +0000 UTC m=+2497.161301698" Jul 29 03:16:53.092598 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:53.092568 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:16:58.140830 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:58.140727 2579 generic.go:356] "Generic (PLEG): container finished" podID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerID="c81c71bc062f9635d785f2f31d470d832e10c00f0c860857ec94db440d96e5f0" exitCode=0 Jul 29 03:16:58.140830 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:58.140800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" event={"ID":"77ec4395-126a-4c03-b30e-256cf34bcae3","Type":"ContainerDied","Data":"c81c71bc062f9635d785f2f31d470d832e10c00f0c860857ec94db440d96e5f0"} Jul 29 03:16:58.141376 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:58.141253 2579 scope.go:117] "RemoveContainer" containerID="c81c71bc062f9635d785f2f31d470d832e10c00f0c860857ec94db440d96e5f0" Jul 29 03:16:58.142366 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:58.142340 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:58.142449 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:58.142409 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:58.142449 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:58.142440 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:16:59.147805 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:59.147760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" event={"ID":"77ec4395-126a-4c03-b30e-256cf34bcae3","Type":"ContainerStarted","Data":"3a0ff861ff384c8ae23560b89f4519ff3bb22beb5caeec97abdd8dbdd3f308b7"} Jul 29 03:16:59.827202 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:59.827159 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:16:59.828403 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:59.828375 2579 patch_prober.go:28] interesting pod/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 29 03:16:59.828530 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:16:59.828446 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 29 03:17:00.057663 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:00.057621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:17:00.826789 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:00.826756 2579 patch_prober.go:28] interesting pod/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 29 03:17:00.827225 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:00.826809 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 29 03:17:01.826878 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:01.826840 2579 patch_prober.go:28] interesting pod/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 29 03:17:01.827309 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:01.826906 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 29 03:17:02.827445 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:02.827401 2579 patch_prober.go:28] interesting pod/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 29 03:17:02.827867 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:02.827468 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 29 03:17:03.831163 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:03.831076 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:17:03.832097 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:17:03.832078 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:20:10.537756 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:20:10.537668 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:20:10.540744 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:20:10.540719 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:25:10.569293 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:10.569265 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:25:10.573608 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:10.573584 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:25:59.373420 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.373385 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7649-140571252956160"] Jul 29 03:25:59.377134 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.377109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7649-140571252956160" Jul 29 03:25:59.379199 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.379172 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-d68hw\"" Jul 29 03:25:59.380604 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.380581 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7649-140571252956160"] Jul 29 03:25:59.411521 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.411493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnz5f\" (UniqueName: \"kubernetes.io/projected/881a6a4f-a1d7-4922-9579-e482533f07f0-kube-api-access-pnz5f\") pod \"test-curl-7649-140571252956160\" (UID: \"881a6a4f-a1d7-4922-9579-e482533f07f0\") " pod="opendatahub/test-curl-7649-140571252956160" Jul 29 03:25:59.512511 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.512470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnz5f\" (UniqueName: \"kubernetes.io/projected/881a6a4f-a1d7-4922-9579-e482533f07f0-kube-api-access-pnz5f\") pod \"test-curl-7649-140571252956160\" (UID: \"881a6a4f-a1d7-4922-9579-e482533f07f0\") " pod="opendatahub/test-curl-7649-140571252956160" Jul 29 03:25:59.520414 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.520382 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnz5f\" (UniqueName: \"kubernetes.io/projected/881a6a4f-a1d7-4922-9579-e482533f07f0-kube-api-access-pnz5f\") pod \"test-curl-7649-140571252956160\" (UID: \"881a6a4f-a1d7-4922-9579-e482533f07f0\") " pod="opendatahub/test-curl-7649-140571252956160" Jul 29 03:25:59.687523 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.687491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7649-140571252956160" Jul 29 03:25:59.821253 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.821218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7649-140571252956160"] Jul 29 03:25:59.823541 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:25:59.823509 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881a6a4f_a1d7_4922_9579_e482533f07f0.slice/crio-74e3382c31136ef207354a6121edddc6454424ac7b43ab5ecdd0dcace6fa984b WatchSource:0}: Error finding container 74e3382c31136ef207354a6121edddc6454424ac7b43ab5ecdd0dcace6fa984b: Status 404 returned error can't find the container with id 74e3382c31136ef207354a6121edddc6454424ac7b43ab5ecdd0dcace6fa984b Jul 29 03:25:59.824904 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.824885 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 03:25:59.837217 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:25:59.837185 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7649-140571252956160" event={"ID":"881a6a4f-a1d7-4922-9579-e482533f07f0","Type":"ContainerStarted","Data":"74e3382c31136ef207354a6121edddc6454424ac7b43ab5ecdd0dcace6fa984b"} Jul 29 03:26:00.843944 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:00.843857 2579 generic.go:356] "Generic (PLEG): container finished" podID="881a6a4f-a1d7-4922-9579-e482533f07f0" containerID="8d8db7db5186f6c1f68403e7dbe38eedae5ba462995a06f88a6016cd26ea480b" exitCode=0 Jul 29 03:26:00.843944 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:00.843910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7649-140571252956160" event={"ID":"881a6a4f-a1d7-4922-9579-e482533f07f0","Type":"ContainerDied","Data":"8d8db7db5186f6c1f68403e7dbe38eedae5ba462995a06f88a6016cd26ea480b"} Jul 29 03:26:01.983927 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:01.983902 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7649-140571252956160" Jul 29 03:26:02.016808 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.016779 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7649-140571252956160_881a6a4f-a1d7-4922-9579-e482533f07f0/test-curl-7649-140571252956160/0.log" Jul 29 03:26:02.031094 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.031067 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnz5f\" (UniqueName: \"kubernetes.io/projected/881a6a4f-a1d7-4922-9579-e482533f07f0-kube-api-access-pnz5f\") pod \"881a6a4f-a1d7-4922-9579-e482533f07f0\" (UID: \"881a6a4f-a1d7-4922-9579-e482533f07f0\") " Jul 29 03:26:02.033036 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.033005 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881a6a4f-a1d7-4922-9579-e482533f07f0-kube-api-access-pnz5f" (OuterVolumeSpecName: "kube-api-access-pnz5f") pod "881a6a4f-a1d7-4922-9579-e482533f07f0" (UID: "881a6a4f-a1d7-4922-9579-e482533f07f0"). InnerVolumeSpecName "kube-api-access-pnz5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:26:02.037179 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.037154 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7649-140571252956160"] Jul 29 03:26:02.038982 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.038895 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7649-140571252956160"] Jul 29 03:26:02.132265 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.132169 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnz5f\" (UniqueName: \"kubernetes.io/projected/881a6a4f-a1d7-4922-9579-e482533f07f0-kube-api-access-pnz5f\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:26:02.452811 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.452776 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881a6a4f-a1d7-4922-9579-e482533f07f0" path="/var/lib/kubelet/pods/881a6a4f-a1d7-4922-9579-e482533f07f0/volumes" Jul 29 03:26:02.855595 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.855498 2579 scope.go:117] "RemoveContainer" containerID="8d8db7db5186f6c1f68403e7dbe38eedae5ba462995a06f88a6016cd26ea480b" Jul 29 03:26:02.855771 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:02.855501 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7649-140571252956160" Jul 29 03:26:05.578769 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.578680 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2"] Jul 29 03:26:05.579228 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.579175 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881a6a4f-a1d7-4922-9579-e482533f07f0" containerName="test-curl-7649-140571252956160" Jul 29 03:26:05.579228 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.579189 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="881a6a4f-a1d7-4922-9579-e482533f07f0" containerName="test-curl-7649-140571252956160" Jul 29 03:26:05.579340 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.579263 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="881a6a4f-a1d7-4922-9579-e482533f07f0" containerName="test-curl-7649-140571252956160" Jul 29 03:26:05.583840 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.583820 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.586370 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.586350 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf2f02-openshift-default-dockercfg-4swt2\"" Jul 29 03:26:05.600526 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.600467 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2"] Jul 29 03:26:05.664879 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.664837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-envoy\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.664929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-podinfo\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.664961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istiod-ca-cert\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.664987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-certs\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.665122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-credential-socket\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.665182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tth\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-kube-api-access-q2tth\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.665213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-token\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665244 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.665234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-data\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.665717 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.665299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-socket\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766134 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-socket\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-envoy\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-podinfo\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istiod-ca-cert\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766308 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-certs\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-credential-socket\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766507 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tth\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-kube-api-access-q2tth\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766612 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-token\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766612 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-data\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766934 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766867 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-certs\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.766934 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-socket\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.767121 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.766947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-credential-socket\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.767185 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.767169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-data\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.767273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.767251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istiod-ca-cert\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.768511 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.768490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-envoy\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.768992 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.768976 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-podinfo\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.773882 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.773862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-token\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.774432 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.774414 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tth\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-kube-api-access-q2tth\") pod \"e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:05.894772 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:05.894669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:06.032650 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.032614 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2"] Jul 29 03:26:06.035754 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:26:06.035725 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dc4c90_69a1_4a4a_8eaf_79740fa4521a.slice/crio-b1a32e8b785624db7027c5e8a9eb40d3e6f7eadfec3be6977ff22e7ff199515d WatchSource:0}: Error finding container b1a32e8b785624db7027c5e8a9eb40d3e6f7eadfec3be6977ff22e7ff199515d: Status 404 returned error can't find the container with id b1a32e8b785624db7027c5e8a9eb40d3e6f7eadfec3be6977ff22e7ff199515d Jul 29 03:26:06.037901 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.037872 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:26:06.037984 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.037951 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:26:06.037984 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.037980 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:26:06.879263 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.879226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" event={"ID":"04dc4c90-69a1-4a4a-8eaf-79740fa4521a","Type":"ContainerStarted","Data":"45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679"} Jul 29 03:26:06.879654 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.879282 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" event={"ID":"04dc4c90-69a1-4a4a-8eaf-79740fa4521a","Type":"ContainerStarted","Data":"b1a32e8b785624db7027c5e8a9eb40d3e6f7eadfec3be6977ff22e7ff199515d"} Jul 29 03:26:06.895383 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.895361 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:06.898149 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:06.898106 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" podStartSLOduration=1.898091534 podStartE2EDuration="1.898091534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:26:06.897066652 +0000 UTC m=+3056.957542038" watchObservedRunningTime="2026-07-29 03:26:06.898091534 +0000 UTC m=+3056.958566919" Jul 29 03:26:07.900137 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:07.900107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:08.889069 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:08.889040 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:08.890089 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:08.890066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:24.931152 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.931118 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 03:26:24.932568 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.931178 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 03:26:24.969107 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.969079 2579 generic.go:356] "Generic (PLEG): container finished" podID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerID="45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679" exitCode=0 Jul 29 03:26:24.969210 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.969146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" event={"ID":"04dc4c90-69a1-4a4a-8eaf-79740fa4521a","Type":"ContainerDied","Data":"45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679"} Jul 29 03:26:24.969562 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.969548 2579 scope.go:117] "RemoveContainer" containerID="45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679" Jul 29 03:26:24.970671 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.970645 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:26:24.970735 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.970712 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:26:24.970770 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:24.970748 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:26:25.979321 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:25.979276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" event={"ID":"04dc4c90-69a1-4a4a-8eaf-79740fa4521a","Type":"ContainerStarted","Data":"a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464"} Jul 29 03:26:26.895625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:26.895574 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:26.895886 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:26.895862 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 29 03:26:26.896049 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:26.895964 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 29 03:26:27.895502 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:27.895460 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 29 03:26:27.895927 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:27.895521 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 29 03:26:28.895460 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:28.895422 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 29 03:26:28.895645 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:28.895482 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 29 03:26:29.898836 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:29.898804 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:29.899370 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:29.899214 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:26:29.899933 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:26:29.899915 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:27:12.621379 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:27:12.621348 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dnxkj_7d17922f-85e0-4da5-96b0-3b7f52be73e3/payload-processing/0.log" Jul 29 03:28:18.209770 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.209743 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dnxkj_7d17922f-85e0-4da5-96b0-3b7f52be73e3/payload-processing/0.log" Jul 29 03:28:18.915216 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.915170 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv"] Jul 29 03:28:18.918727 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.918706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.920816 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.920793 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-002e42-openshift-default-dockercfg-whzrh\"" Jul 29 03:28:18.920930 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.920862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-credential-socket\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.920930 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.920899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-data\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.921056 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.920954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-podinfo\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.921056 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.921002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-token\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.921126 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.921114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-envoy\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.921180 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.921170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-certs\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.921298 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.921191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-socket\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.921298 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.921227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e92199-ffd0-45c1-8108-6e91afb963b3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.921298 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.921248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnh4m\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-kube-api-access-lnh4m\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:18.931372 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:18.931347 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv"] Jul 29 03:28:19.021790 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.021756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-certs\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022044 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-socket\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022220 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e92199-ffd0-45c1-8108-6e91afb963b3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022332 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnh4m\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-kube-api-access-lnh4m\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022497 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-credential-socket\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022615 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-data\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022728 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-podinfo\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022828 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-token\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.022967 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.022953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-envoy\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.024323 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.024295 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-certs\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.024707 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.024689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-socket\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.025365 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.025336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-data\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.025615 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.025593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-credential-socket\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.025776 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.025756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e92199-ffd0-45c1-8108-6e91afb963b3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.028824 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.028800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-envoy\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.035119 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.035097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-podinfo\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.045385 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.044495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-token\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.045385 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.045340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnh4m\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-kube-api-access-lnh4m\") pod \"e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.229813 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.229775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:19.380273 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.380219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv"] Jul 29 03:28:19.382743 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:28:19.382717 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e92199_ffd0_45c1_8108_6e91afb963b3.slice/crio-c1f290fbfeb0df0c74ce4927b042373a45f419ff71a919384fc23d36c6967d85 WatchSource:0}: Error finding container c1f290fbfeb0df0c74ce4927b042373a45f419ff71a919384fc23d36c6967d85: Status 404 returned error can't find the container with id c1f290fbfeb0df0c74ce4927b042373a45f419ff71a919384fc23d36c6967d85 Jul 29 03:28:19.385074 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.385040 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:28:19.385148 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.385109 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:28:19.385148 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.385137 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:28:19.552515 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.552381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" event={"ID":"c4e92199-ffd0-45c1-8108-6e91afb963b3","Type":"ContainerStarted","Data":"3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f"} Jul 29 03:28:19.552515 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.552454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" event={"ID":"c4e92199-ffd0-45c1-8108-6e91afb963b3","Type":"ContainerStarted","Data":"c1f290fbfeb0df0c74ce4927b042373a45f419ff71a919384fc23d36c6967d85"} Jul 29 03:28:19.572285 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:19.572225 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" podStartSLOduration=1.572208353 podStartE2EDuration="1.572208353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:28:19.57135526 +0000 UTC m=+3189.631830644" watchObservedRunningTime="2026-07-29 03:28:19.572208353 +0000 UTC m=+3189.632683738" Jul 29 03:28:20.229975 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:20.229933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:20.234990 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:20.234965 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:20.558900 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:20.558815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:20.560101 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:20.560080 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:34.631262 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:34.631230 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerID="3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f" exitCode=0 Jul 29 03:28:34.631648 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:34.631306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" event={"ID":"c4e92199-ffd0-45c1-8108-6e91afb963b3","Type":"ContainerDied","Data":"3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f"} Jul 29 03:28:34.631817 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:34.631802 2579 scope.go:117] "RemoveContainer" containerID="3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f" Jul 29 03:28:34.632981 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:34.632954 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:28:34.633161 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:34.633033 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:28:34.633161 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:34.633068 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 03:28:35.559480 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:35.559406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:35.637751 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:35.637708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" event={"ID":"c4e92199-ffd0-45c1-8108-6e91afb963b3","Type":"ContainerStarted","Data":"8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773"} Jul 29 03:28:36.229936 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:36.229899 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:36.231065 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:36.231035 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 29 03:28:36.231193 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:36.231084 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 29 03:28:37.230761 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:37.230721 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 29 03:28:37.231208 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:37.230779 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 29 03:28:38.230800 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:38.230757 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 29 03:28:38.231215 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:38.230819 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 29 03:28:39.231007 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:39.230965 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 29 03:28:39.231447 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:39.231060 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 29 03:28:40.234616 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:40.234585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:40.235027 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:40.234866 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:40.235518 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:40.235500 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:28:58.811436 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:58.811394 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv"] Jul 29 03:28:58.812000 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:28:58.811707 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" containerID="cri-o://8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773" gracePeriod=30 Jul 29 03:29:04.068082 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.068055 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:29:04.174524 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174435 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e92199-ffd0-45c1-8108-6e91afb963b3-istiod-ca-cert\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.174524 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174502 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-credential-socket\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.174759 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-podinfo\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.174759 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174644 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnh4m\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-kube-api-access-lnh4m\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.174759 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-envoy\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.174759 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174721 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-token\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.174759 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174751 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-socket\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.175053 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174809 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-data\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.175053 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174807 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:29:04.175053 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174873 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e92199-ffd0-45c1-8108-6e91afb963b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:29:04.175053 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.174910 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-certs\") pod \"c4e92199-ffd0-45c1-8108-6e91afb963b3\" (UID: \"c4e92199-ffd0-45c1-8108-6e91afb963b3\") " Jul 29 03:29:04.175270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.175177 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:29:04.175347 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.175317 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e92199-ffd0-45c1-8108-6e91afb963b3-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.175409 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.175343 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.175409 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.175359 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.175589 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.175566 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:29:04.176515 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.176487 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:29:04.176967 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.176932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:29:04.177178 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.177157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-kube-api-access-lnh4m" (OuterVolumeSpecName: "kube-api-access-lnh4m") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "kube-api-access-lnh4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:29:04.177240 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.177157 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:29:04.177278 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.177235 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c4e92199-ffd0-45c1-8108-6e91afb963b3" (UID: "c4e92199-ffd0-45c1-8108-6e91afb963b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:29:04.276355 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.276317 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.276355 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.276350 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnh4m\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-kube-api-access-lnh4m\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.276355 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.276362 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.276606 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.276370 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.276606 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.276380 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.276606 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.276387 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e92199-ffd0-45c1-8108-6e91afb963b3-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:29:04.789599 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.789494 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerID="8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773" exitCode=0 Jul 29 03:29:04.789599 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.789537 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" Jul 29 03:29:04.789599 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.789572 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" event={"ID":"c4e92199-ffd0-45c1-8108-6e91afb963b3","Type":"ContainerDied","Data":"8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773"} Jul 29 03:29:04.789875 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.789632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv" event={"ID":"c4e92199-ffd0-45c1-8108-6e91afb963b3","Type":"ContainerDied","Data":"c1f290fbfeb0df0c74ce4927b042373a45f419ff71a919384fc23d36c6967d85"} Jul 29 03:29:04.789875 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.789661 2579 scope.go:117] "RemoveContainer" containerID="8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773" Jul 29 03:29:04.798518 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.798488 2579 scope.go:117] "RemoveContainer" containerID="3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f" Jul 29 03:29:04.806905 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.806880 2579 scope.go:117] "RemoveContainer" containerID="8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773" Jul 29 03:29:04.807186 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:29:04.807167 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773\": container with ID starting with 8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773 not found: ID does not exist" containerID="8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773" Jul 29 03:29:04.807249 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.807198 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773"} err="failed to get container status \"8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773\": rpc error: code = NotFound desc = could not find container \"8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773\": container with ID starting with 8e01474bf49b36501cc445d6a023832fa35d2517ca391fe443be6bf208217773 not found: ID does not exist" Jul 29 03:29:04.807249 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.807216 2579 scope.go:117] "RemoveContainer" containerID="3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f" Jul 29 03:29:04.807346 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.807270 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv"] Jul 29 03:29:04.807466 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:29:04.807449 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f\": container with ID starting with 3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f not found: ID does not exist" containerID="3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f" Jul 29 03:29:04.807508 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.807473 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f"} err="failed to get container status \"3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f\": rpc error: code = NotFound desc = could not find container \"3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f\": container with ID starting with 3c2b431a776ead4bc45c75f794fd31d21ce75beb32d61731020c48ccc7e5fd1f not found: ID does not exist" Jul 29 03:29:04.811100 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:04.811077 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-002e42-openshift-default-54f5f975c8-84jgv"] Jul 29 03:29:06.453035 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:29:06.452981 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" path="/var/lib/kubelet/pods/c4e92199-ffd0-45c1-8108-6e91afb963b3/volumes" Jul 29 03:30:10.598808 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:10.598780 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:30:10.604697 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:10.604675 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:30:12.081337 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:12.081304 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2"] Jul 29 03:30:12.081784 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:12.081537 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" containerID="cri-o://a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464" gracePeriod=30 Jul 29 03:30:14.901813 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:14.901772 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 03:30:14.902303 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:14.901847 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:30:17.340984 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.340953 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:30:17.404151 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404056 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istiod-ca-cert\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404151 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-data\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404151 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404129 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-podinfo\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404389 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404193 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-envoy\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404389 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404220 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2tth\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-kube-api-access-q2tth\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404389 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-certs\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404389 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-credential-socket\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404389 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404371 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-token\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404645 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-socket\") pod \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\" (UID: \"04dc4c90-69a1-4a4a-8eaf-79740fa4521a\") " Jul 29 03:30:17.404645 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404531 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:30:17.404829 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404775 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.404829 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404779 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:17.404829 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404817 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:17.404997 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404826 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:17.404997 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.404890 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-data" (OuterVolumeSpecName: "istio-data") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:17.406596 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.406573 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:30:17.406689 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.406624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-token" (OuterVolumeSpecName: "istio-token") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:30:17.406812 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.406790 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:17.406872 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.406828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-kube-api-access-q2tth" (OuterVolumeSpecName: "kube-api-access-q2tth") pod "04dc4c90-69a1-4a4a-8eaf-79740fa4521a" (UID: "04dc4c90-69a1-4a4a-8eaf-79740fa4521a"). InnerVolumeSpecName "kube-api-access-q2tth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:30:17.506301 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506262 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.506301 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506296 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.506301 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506305 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.506301 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506312 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.506590 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506322 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2tth\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-kube-api-access-q2tth\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.506590 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506331 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.506590 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506339 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:17.506590 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:17.506349 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dc4c90-69a1-4a4a-8eaf-79740fa4521a-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:18.169658 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.169622 2579 generic.go:356] "Generic (PLEG): container finished" podID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerID="a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464" exitCode=0 Jul 29 03:30:18.169882 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.169679 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" Jul 29 03:30:18.169882 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.169705 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" event={"ID":"04dc4c90-69a1-4a4a-8eaf-79740fa4521a","Type":"ContainerDied","Data":"a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464"} Jul 29 03:30:18.169882 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.169768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2" event={"ID":"04dc4c90-69a1-4a4a-8eaf-79740fa4521a","Type":"ContainerDied","Data":"b1a32e8b785624db7027c5e8a9eb40d3e6f7eadfec3be6977ff22e7ff199515d"} Jul 29 03:30:18.169882 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.169790 2579 scope.go:117] "RemoveContainer" containerID="a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464" Jul 29 03:30:18.179299 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.179277 2579 scope.go:117] "RemoveContainer" containerID="45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679" Jul 29 03:30:18.191349 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.191193 2579 scope.go:117] "RemoveContainer" containerID="a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464" Jul 29 03:30:18.191695 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.191673 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2"] Jul 29 03:30:18.191791 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:30:18.191733 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464\": container with ID starting with a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464 not found: ID does not exist" containerID="a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464" Jul 29 03:30:18.191791 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.191765 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464"} err="failed to get container status \"a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464\": rpc error: code = NotFound desc = could not find container \"a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464\": container with ID starting with a36d243a8476909ad64e05e5307d7d7a2da360233a508d775bfe01863c0ef464 not found: ID does not exist" Jul 29 03:30:18.191791 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.191784 2579 scope.go:117] "RemoveContainer" containerID="45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679" Jul 29 03:30:18.192142 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:30:18.192119 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679\": container with ID starting with 45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679 not found: ID does not exist" containerID="45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679" Jul 29 03:30:18.192218 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.192147 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679"} err="failed to get container status \"45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679\": rpc error: code = NotFound desc = could not find container \"45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679\": container with ID starting with 45e8ff4a8b64837057c8069e2a5ae925083d479ba76d6e976f7d47c1b979a679 not found: ID does not exist" Jul 29 03:30:18.193750 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.193733 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf2f02-openshift-default-647c7f868b-wpsf2"] Jul 29 03:30:18.452825 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:18.452792 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" path="/var/lib/kubelet/pods/04dc4c90-69a1-4a4a-8eaf-79740fa4521a/volumes" Jul 29 03:30:40.106625 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.106585 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9"] Jul 29 03:30:40.107154 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.106797 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" podUID="8b53d6c4-9fb2-43b6-996d-e5195c418824" containerName="maas-api" containerID="cri-o://dd8118e0ceee1fcd293171adadc5b4785fce2187f841a636c0f22ba84445e047" gracePeriod=30 Jul 29 03:30:40.286126 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.286095 2579 generic.go:356] "Generic (PLEG): container finished" podID="8b53d6c4-9fb2-43b6-996d-e5195c418824" containerID="dd8118e0ceee1fcd293171adadc5b4785fce2187f841a636c0f22ba84445e047" exitCode=0 Jul 29 03:30:40.286278 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.286172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" event={"ID":"8b53d6c4-9fb2-43b6-996d-e5195c418824","Type":"ContainerDied","Data":"dd8118e0ceee1fcd293171adadc5b4785fce2187f841a636c0f22ba84445e047"} Jul 29 03:30:40.387388 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.387361 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:30:40.535306 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.535272 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lltx\" (UniqueName: \"kubernetes.io/projected/8b53d6c4-9fb2-43b6-996d-e5195c418824-kube-api-access-8lltx\") pod \"8b53d6c4-9fb2-43b6-996d-e5195c418824\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " Jul 29 03:30:40.535496 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.535323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls\") pod \"8b53d6c4-9fb2-43b6-996d-e5195c418824\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " Jul 29 03:30:40.535496 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.535428 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b53d6c4-9fb2-43b6-996d-e5195c418824-openshift-service-ca\") pod \"8b53d6c4-9fb2-43b6-996d-e5195c418824\" (UID: \"8b53d6c4-9fb2-43b6-996d-e5195c418824\") " Jul 29 03:30:40.535817 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.535795 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b53d6c4-9fb2-43b6-996d-e5195c418824-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b53d6c4-9fb2-43b6-996d-e5195c418824" (UID: "8b53d6c4-9fb2-43b6-996d-e5195c418824"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:30:40.537500 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.537477 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b53d6c4-9fb2-43b6-996d-e5195c418824" (UID: "8b53d6c4-9fb2-43b6-996d-e5195c418824"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:30:40.537593 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.537577 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b53d6c4-9fb2-43b6-996d-e5195c418824-kube-api-access-8lltx" (OuterVolumeSpecName: "kube-api-access-8lltx") pod "8b53d6c4-9fb2-43b6-996d-e5195c418824" (UID: "8b53d6c4-9fb2-43b6-996d-e5195c418824"). InnerVolumeSpecName "kube-api-access-8lltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:30:40.636513 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.636422 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b53d6c4-9fb2-43b6-996d-e5195c418824-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:40.636513 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.636454 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lltx\" (UniqueName: \"kubernetes.io/projected/8b53d6c4-9fb2-43b6-996d-e5195c418824-kube-api-access-8lltx\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:40.636513 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:40.636466 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b53d6c4-9fb2-43b6-996d-e5195c418824-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:41.292737 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:41.292697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" event={"ID":"8b53d6c4-9fb2-43b6-996d-e5195c418824","Type":"ContainerDied","Data":"82653cf2f76f668cd2af67de12ed62db9210e1b96dfe8a768162e41605df16d3"} Jul 29 03:30:41.293183 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:41.292764 2579 scope.go:117] "RemoveContainer" containerID="dd8118e0ceee1fcd293171adadc5b4785fce2187f841a636c0f22ba84445e047" Jul 29 03:30:41.293183 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:41.292710 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9" Jul 29 03:30:41.313489 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:41.313454 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9"] Jul 29 03:30:41.316851 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:41.316821 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f4870d-5bf9c48987-4tgt9"] Jul 29 03:30:42.453677 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:42.453643 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b53d6c4-9fb2-43b6-996d-e5195c418824" path="/var/lib/kubelet/pods/8b53d6c4-9fb2-43b6-996d-e5195c418824/volumes" Jul 29 03:30:46.730464 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:46.730424 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8"] Jul 29 03:30:46.730909 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:46.730693 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" containerID="cri-o://c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8" gracePeriod=30 Jul 29 03:30:51.169460 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:51.169419 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 03:30:51.170006 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:51.169519 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 03:30:52.001301 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.001232 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:30:52.037068 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037026 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41093fff-8613-4fa9-87ad-10e0f6b194e5-istiod-ca-cert\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037083 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-data\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-podinfo\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037185 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-socket\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037274 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037218 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-token\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037566 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037461 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41093fff-8613-4fa9-87ad-10e0f6b194e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:30:52.037566 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037547 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:52.037723 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-certs\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037779 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-credential-socket\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037835 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037782 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-envoy\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037835 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037811 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw86f\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-kube-api-access-hw86f\") pod \"41093fff-8613-4fa9-87ad-10e0f6b194e5\" (UID: \"41093fff-8613-4fa9-87ad-10e0f6b194e5\") " Jul 29 03:30:52.037931 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.037900 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:52.038302 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.038228 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41093fff-8613-4fa9-87ad-10e0f6b194e5-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.038302 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.038233 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:52.038302 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.038253 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.038302 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.038268 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.038302 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.038260 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:52.039744 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.039715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:30:52.040035 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.039994 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:30:52.040146 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.040090 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-kube-api-access-hw86f" (OuterVolumeSpecName: "kube-api-access-hw86f") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "kube-api-access-hw86f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:30:52.040215 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.040189 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41093fff-8613-4fa9-87ad-10e0f6b194e5" (UID: "41093fff-8613-4fa9-87ad-10e0f6b194e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:30:52.138921 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.138877 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.138921 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.138912 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.138921 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.138925 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw86f\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-kube-api-access-hw86f\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.139219 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.138936 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.139219 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.138948 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.139219 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.138959 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41093fff-8613-4fa9-87ad-10e0f6b194e5-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:52.356810 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.356716 2579 generic.go:356] "Generic (PLEG): container finished" podID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerID="c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8" exitCode=0 Jul 29 03:30:52.356810 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.356764 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" Jul 29 03:30:52.357281 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.356768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" event={"ID":"41093fff-8613-4fa9-87ad-10e0f6b194e5","Type":"ContainerDied","Data":"c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8"} Jul 29 03:30:52.357281 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.356876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8" event={"ID":"41093fff-8613-4fa9-87ad-10e0f6b194e5","Type":"ContainerDied","Data":"b83a5ff2dace618a566f69243136de11e0e40aff0bb28e86ddaf5a07b4945873"} Jul 29 03:30:52.357281 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.356900 2579 scope.go:117] "RemoveContainer" containerID="c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8" Jul 29 03:30:52.366915 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.366896 2579 scope.go:117] "RemoveContainer" containerID="c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a" Jul 29 03:30:52.375048 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.375027 2579 scope.go:117] "RemoveContainer" containerID="c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8" Jul 29 03:30:52.375423 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:30:52.375399 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8\": container with ID starting with c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8 not found: ID does not exist" containerID="c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8" Jul 29 03:30:52.375510 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.375432 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8"} err="failed to get container status \"c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8\": rpc error: code = NotFound desc = could not find container \"c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8\": container with ID starting with c65eb4b2aee4014ef3dbbc62ec866a1fa0f1541c2b0135c3cbfc6dd356f6c1b8 not found: ID does not exist" Jul 29 03:30:52.375510 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.375450 2579 scope.go:117] "RemoveContainer" containerID="c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a" Jul 29 03:30:52.375783 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:30:52.375766 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a\": container with ID starting with c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a not found: ID does not exist" containerID="c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a" Jul 29 03:30:52.375845 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.375788 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a"} err="failed to get container status \"c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a\": rpc error: code = NotFound desc = could not find container \"c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a\": container with ID starting with c38d4fc6a3a49e852c86af1e725eb8ff7ce1a305ae76117dd7ef00a73fb6031a not found: ID does not exist" Jul 29 03:30:52.378641 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.378619 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8"] Jul 29 03:30:52.383231 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.383210 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-f4870d-openshift-default-7b6d59dfbd-fwsx8"] Jul 29 03:30:52.453520 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:52.453487 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" path="/var/lib/kubelet/pods/41093fff-8613-4fa9-87ad-10e0f6b194e5/volumes" Jul 29 03:30:57.637060 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.637010 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7"] Jul 29 03:30:57.637629 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.637310 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" podUID="50a347c3-94ee-4cd4-9bc0-90c292316de8" containerName="maas-api" containerID="cri-o://0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280" gracePeriod=30 Jul 29 03:30:57.906563 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.906493 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:30:57.993354 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.993319 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a347c3-94ee-4cd4-9bc0-90c292316de8-maas-api-tls\") pod \"50a347c3-94ee-4cd4-9bc0-90c292316de8\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " Jul 29 03:30:57.993534 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.993405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csrcz\" (UniqueName: \"kubernetes.io/projected/50a347c3-94ee-4cd4-9bc0-90c292316de8-kube-api-access-csrcz\") pod \"50a347c3-94ee-4cd4-9bc0-90c292316de8\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " Jul 29 03:30:57.993534 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.993451 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a347c3-94ee-4cd4-9bc0-90c292316de8-openshift-service-ca\") pod \"50a347c3-94ee-4cd4-9bc0-90c292316de8\" (UID: \"50a347c3-94ee-4cd4-9bc0-90c292316de8\") " Jul 29 03:30:57.993869 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.993842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a347c3-94ee-4cd4-9bc0-90c292316de8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50a347c3-94ee-4cd4-9bc0-90c292316de8" (UID: "50a347c3-94ee-4cd4-9bc0-90c292316de8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:30:57.995451 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.995416 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a347c3-94ee-4cd4-9bc0-90c292316de8-kube-api-access-csrcz" (OuterVolumeSpecName: "kube-api-access-csrcz") pod "50a347c3-94ee-4cd4-9bc0-90c292316de8" (UID: "50a347c3-94ee-4cd4-9bc0-90c292316de8"). InnerVolumeSpecName "kube-api-access-csrcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:30:57.995545 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:57.995492 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a347c3-94ee-4cd4-9bc0-90c292316de8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50a347c3-94ee-4cd4-9bc0-90c292316de8" (UID: "50a347c3-94ee-4cd4-9bc0-90c292316de8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 03:30:58.094678 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.094640 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a347c3-94ee-4cd4-9bc0-90c292316de8-maas-api-tls\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:58.094678 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.094672 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrcz\" (UniqueName: \"kubernetes.io/projected/50a347c3-94ee-4cd4-9bc0-90c292316de8-kube-api-access-csrcz\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:58.094678 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.094683 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a347c3-94ee-4cd4-9bc0-90c292316de8-openshift-service-ca\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:30:58.388341 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.388300 2579 generic.go:356] "Generic (PLEG): container finished" podID="50a347c3-94ee-4cd4-9bc0-90c292316de8" containerID="0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280" exitCode=0 Jul 29 03:30:58.388341 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.388342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" event={"ID":"50a347c3-94ee-4cd4-9bc0-90c292316de8","Type":"ContainerDied","Data":"0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280"} Jul 29 03:30:58.388867 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.388350 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" Jul 29 03:30:58.388867 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.388385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7" event={"ID":"50a347c3-94ee-4cd4-9bc0-90c292316de8","Type":"ContainerDied","Data":"41c963aecbe4171424822115bdb3f3b6a0176c8038c8ab7b9a49b2c0ae71b948"} Jul 29 03:30:58.388867 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.388410 2579 scope.go:117] "RemoveContainer" containerID="0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280" Jul 29 03:30:58.397401 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.397382 2579 scope.go:117] "RemoveContainer" containerID="0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280" Jul 29 03:30:58.397657 ip-10-0-141-222 kubenswrapper[2579]: E0729 03:30:58.397636 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280\": container with ID starting with 0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280 not found: ID does not exist" containerID="0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280" Jul 29 03:30:58.397718 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.397668 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280"} err="failed to get container status \"0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280\": rpc error: code = NotFound desc = could not find container \"0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280\": container with ID starting with 0a5c9e090b5d2f4574ca3a9bf313e8527ac0975aa6878808d02b2847e3446280 not found: ID does not exist" Jul 29 03:30:58.408478 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.408451 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7"] Jul 29 03:30:58.412779 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.412752 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-19184d-f5495f88d-l9lj7"] Jul 29 03:30:58.452884 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:30:58.452850 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a347c3-94ee-4cd4-9bc0-90c292316de8" path="/var/lib/kubelet/pods/50a347c3-94ee-4cd4-9bc0-90c292316de8/volumes" Jul 29 03:31:04.267520 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:04.267485 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh"] Jul 29 03:31:04.268031 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:04.267770 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" containerID="cri-o://3a0ff861ff384c8ae23560b89f4519ff3bb22beb5caeec97abdd8dbdd3f308b7" gracePeriod=30 Jul 29 03:31:04.936099 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:04.936065 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c5488cb64-smstx_499fd96b-aced-4e44-95e1-5b81f206a24e/authorino/0.log" Jul 29 03:31:09.443868 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.443837 2579 generic.go:356] "Generic (PLEG): container finished" podID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerID="3a0ff861ff384c8ae23560b89f4519ff3bb22beb5caeec97abdd8dbdd3f308b7" exitCode=0 Jul 29 03:31:09.444322 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.443928 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" event={"ID":"77ec4395-126a-4c03-b30e-256cf34bcae3","Type":"ContainerDied","Data":"3a0ff861ff384c8ae23560b89f4519ff3bb22beb5caeec97abdd8dbdd3f308b7"} Jul 29 03:31:09.444322 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.444043 2579 scope.go:117] "RemoveContainer" containerID="c81c71bc062f9635d785f2f31d470d832e10c00f0c860857ec94db440d96e5f0" Jul 29 03:31:09.535925 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.535856 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:31:09.606105 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606070 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-certs\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606105 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-token\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606335 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606145 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-credential-socket\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606335 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c85p5\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-kube-api-access-c85p5\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606335 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606221 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-socket\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606335 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606254 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ec4395-126a-4c03-b30e-256cf34bcae3-istiod-ca-cert\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606335 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606289 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-podinfo\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606335 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606331 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-envoy\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606627 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606357 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-data\") pod \"77ec4395-126a-4c03-b30e-256cf34bcae3\" (UID: \"77ec4395-126a-4c03-b30e-256cf34bcae3\") " Jul 29 03:31:09.606627 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606412 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:31:09.606627 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606478 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:31:09.606627 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606578 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:31:09.606828 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606748 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-certs\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.606828 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606770 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-credential-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.606828 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606785 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-workload-socket\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.606951 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.606934 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ec4395-126a-4c03-b30e-256cf34bcae3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 03:31:09.607357 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.607326 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-data" (OuterVolumeSpecName: "istio-data") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:31:09.608491 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.608465 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-kube-api-access-c85p5" (OuterVolumeSpecName: "kube-api-access-c85p5") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "kube-api-access-c85p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:31:09.608580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.608496 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-token" (OuterVolumeSpecName: "istio-token") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 03:31:09.608580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.608542 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 03:31:09.608645 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.608586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77ec4395-126a-4c03-b30e-256cf34bcae3" (UID: "77ec4395-126a-4c03-b30e-256cf34bcae3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 03:31:09.707508 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.707463 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-podinfo\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.707508 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.707497 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-envoy\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.707508 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.707506 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-data\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.707508 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.707513 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-istio-token\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.707508 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.707521 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c85p5\" (UniqueName: \"kubernetes.io/projected/77ec4395-126a-4c03-b30e-256cf34bcae3-kube-api-access-c85p5\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:09.707871 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:09.707529 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ec4395-126a-4c03-b30e-256cf34bcae3-istiod-ca-cert\") on node \"ip-10-0-141-222.ec2.internal\" DevicePath \"\"" Jul 29 03:31:10.315965 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.315927 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7trrb_8f72f594-e7cc-4154-a777-77ba264f82f6/manager/0.log" Jul 29 03:31:10.442202 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.442170 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b7rjr_ebe6d1fa-9def-431b-8092-0ac85bdfe965/manager/0.log" Jul 29 03:31:10.451446 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.451420 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" Jul 29 03:31:10.453110 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.453084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh" event={"ID":"77ec4395-126a-4c03-b30e-256cf34bcae3","Type":"ContainerDied","Data":"cfec96fbe04936113a706ca54aebd436850bb9db6f43081e2ddc97ac32edd6a8"} Jul 29 03:31:10.453207 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.453133 2579 scope.go:117] "RemoveContainer" containerID="3a0ff861ff384c8ae23560b89f4519ff3bb22beb5caeec97abdd8dbdd3f308b7" Jul 29 03:31:10.475187 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.475149 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh"] Jul 29 03:31:10.479541 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.479512 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-19184d-openshift-default-68fdcc88cb-jgqnh"] Jul 29 03:31:10.570928 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.570838 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b54d569bf-pwmlg_81407d25-a23e-4c4f-8b86-59cc4bf3c539/manager/0.log" Jul 29 03:31:10.686393 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.686361 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-q6tnz_440221a6-cc0f-46a5-afe8-da54477362df/server/0.log" Jul 29 03:31:10.816240 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.816213 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2jldv_d8abdbca-5306-4318-acbc-bdbc6aef21a2/manager/0.log" Jul 29 03:31:10.932959 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.932929 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dbd45849d-857zf_4187b9e7-bba0-4ff9-a987-0179b32924e0/copy-manifests/0.log" Jul 29 03:31:10.939443 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:10.939416 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dbd45849d-857zf_4187b9e7-bba0-4ff9-a987-0179b32924e0/manager/0.log" Jul 29 03:31:11.568024 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:11.567979 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58d5b6f47b-vg2zh_53e82640-b1dd-4d27-acdb-7b365fef14fc/maas-api/0.log" Jul 29 03:31:12.036580 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:12.036542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2kncg_21303fd6-e73c-42b4-bbbc-5f0a0cff26d2/postgres/0.log" Jul 29 03:31:12.452537 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:12.452502 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" path="/var/lib/kubelet/pods/77ec4395-126a-4c03-b30e-256cf34bcae3/volumes" Jul 29 03:31:13.315694 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:13.315606 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c5488cb64-smstx_499fd96b-aced-4e44-95e1-5b81f206a24e/authorino/0.log" Jul 29 03:31:13.439606 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:13.439573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dc4c6_b7206433-81e2-4331-9650-33570a02e3e8/manager/0.log" Jul 29 03:31:13.551201 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:13.551174 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qqpwv_5a1ff45b-4a0b-441a-b112-411303f26976/manager/0.log" Jul 29 03:31:13.788430 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:13.788397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4b9kw_790610a4-7db1-420f-b481-a41d41a828f0/registry-server/0.log" Jul 29 03:31:13.911319 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:13.911288 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pzkxr_63a5d434-45db-443a-9c54-97a01baab316/manager/0.log" Jul 29 03:31:14.484325 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:14.484296 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb_f3bd5a2c-bde0-4eb9-9308-c69a6eae8357/istio-proxy/0.log" Jul 29 03:31:14.821294 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:14.821209 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdc76d774-lmbzn_6ebef28c-7b32-4b43-a0de-4a197da67ef5/kube-auth-proxy/0.log" Jul 29 03:31:14.949736 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:14.949707 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k5qpd_379ad44b-8b7d-4957-a529-1be09e159bbc/istio-proxy/1.log" Jul 29 03:31:15.201897 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:15.201845 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dnxkj_7d17922f-85e0-4da5-96b0-3b7f52be73e3/payload-processing/0.log" Jul 29 03:31:15.921175 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:15.921143 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ttcf5_134849eb-a2f3-412a-88b8-4719949b0c2f/main/0.log" Jul 29 03:31:22.075357 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075316 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2j9bk/must-gather-h2r9r"] Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075682 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075693 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075702 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075707 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075717 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075722 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075730 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075735 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075741 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075746 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075756 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075761 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075767 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a347c3-94ee-4cd4-9bc0-90c292316de8" containerName="maas-api" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075771 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a347c3-94ee-4cd4-9bc0-90c292316de8" containerName="maas-api" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075779 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075786 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075792 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b53d6c4-9fb2-43b6-996d-e5195c418824" containerName="maas-api" Jul 29 03:31:22.075819 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075797 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b53d6c4-9fb2-43b6-996d-e5195c418824" containerName="maas-api" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075855 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075864 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ec4395-126a-4c03-b30e-256cf34bcae3" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075869 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075875 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075881 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075886 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075892 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41093fff-8613-4fa9-87ad-10e0f6b194e5" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075898 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a347c3-94ee-4cd4-9bc0-90c292316de8" containerName="maas-api" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075905 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e92199-ffd0-45c1-8108-6e91afb963b3" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075910 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b53d6c4-9fb2-43b6-996d-e5195c418824" containerName="maas-api" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075966 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" Jul 29 03:31:22.076468 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.075971 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04dc4c90-69a1-4a4a-8eaf-79740fa4521a" containerName="istio-proxy" Jul 29 03:31:22.080269 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.080245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.082403 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.082377 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2j9bk\"/\"openshift-service-ca.crt\"" Jul 29 03:31:22.082883 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.082865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2j9bk\"/\"default-dockercfg-mbphh\"" Jul 29 03:31:22.082971 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.082887 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2j9bk\"/\"kube-root-ca.crt\"" Jul 29 03:31:22.087502 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.087463 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2j9bk/must-gather-h2r9r"] Jul 29 03:31:22.216771 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.216733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54596276-5b9d-4afc-b020-2738d0fff523-must-gather-output\") pod \"must-gather-h2r9r\" (UID: \"54596276-5b9d-4afc-b020-2738d0fff523\") " pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.216954 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.216871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg4hn\" (UniqueName: \"kubernetes.io/projected/54596276-5b9d-4afc-b020-2738d0fff523-kube-api-access-fg4hn\") pod \"must-gather-h2r9r\" (UID: \"54596276-5b9d-4afc-b020-2738d0fff523\") " pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.317896 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.317857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg4hn\" (UniqueName: \"kubernetes.io/projected/54596276-5b9d-4afc-b020-2738d0fff523-kube-api-access-fg4hn\") pod \"must-gather-h2r9r\" (UID: \"54596276-5b9d-4afc-b020-2738d0fff523\") " pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.318102 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.317906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54596276-5b9d-4afc-b020-2738d0fff523-must-gather-output\") pod \"must-gather-h2r9r\" (UID: \"54596276-5b9d-4afc-b020-2738d0fff523\") " pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.318237 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.318221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54596276-5b9d-4afc-b020-2738d0fff523-must-gather-output\") pod \"must-gather-h2r9r\" (UID: \"54596276-5b9d-4afc-b020-2738d0fff523\") " pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.325757 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.325684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg4hn\" (UniqueName: \"kubernetes.io/projected/54596276-5b9d-4afc-b020-2738d0fff523-kube-api-access-fg4hn\") pod \"must-gather-h2r9r\" (UID: \"54596276-5b9d-4afc-b020-2738d0fff523\") " pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.390322 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.390279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2j9bk/must-gather-h2r9r" Jul 29 03:31:22.526499 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.526473 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2j9bk/must-gather-h2r9r"] Jul 29 03:31:22.529072 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:31:22.529036 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54596276_5b9d_4afc_b020_2738d0fff523.slice/crio-180a946cd7e71f4b10684f4c3c35d99f8ffa6c2ae04f9826dd25ca12eb54c68c WatchSource:0}: Error finding container 180a946cd7e71f4b10684f4c3c35d99f8ffa6c2ae04f9826dd25ca12eb54c68c: Status 404 returned error can't find the container with id 180a946cd7e71f4b10684f4c3c35d99f8ffa6c2ae04f9826dd25ca12eb54c68c Jul 29 03:31:22.531098 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:22.531083 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 03:31:23.513524 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:23.513495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2j9bk/must-gather-h2r9r" event={"ID":"54596276-5b9d-4afc-b020-2738d0fff523","Type":"ContainerStarted","Data":"180a946cd7e71f4b10684f4c3c35d99f8ffa6c2ae04f9826dd25ca12eb54c68c"} Jul 29 03:31:24.522493 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:24.522447 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2j9bk/must-gather-h2r9r" event={"ID":"54596276-5b9d-4afc-b020-2738d0fff523","Type":"ContainerStarted","Data":"5b48176ae2e4eabd191700cd1e8f9d4a0d403271adb4ab9bdf8e42826e3f01b0"} Jul 29 03:31:24.522984 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:24.522507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2j9bk/must-gather-h2r9r" event={"ID":"54596276-5b9d-4afc-b020-2738d0fff523","Type":"ContainerStarted","Data":"ca026de3ea5418b43d64f316f239eb65d39d4b31bd470761561c7f84931b8bde"} Jul 29 03:31:24.537775 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:24.537686 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2j9bk/must-gather-h2r9r" podStartSLOduration=1.607763881 podStartE2EDuration="2.537662911s" totalImagesPullingTime="929.89903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 03:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:31:24.536059653 +0000 UTC m=+3374.596535039" watchObservedRunningTime="2026-07-29 03:31:24.537662911 +0000 UTC m=+3374.598138296" Jul 29 03:31:25.011301 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:25.011266 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2rsk8_dc64185a-12f2-4b53-9f35-a67980be08bb/global-pull-secret-syncer/0.log" Jul 29 03:31:25.213990 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:25.213951 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zmlzv_128f1d0e-50e5-4717-8531-dd5aa754433d/konnectivity-agent/0.log" Jul 29 03:31:25.281175 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:25.281095 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-222.ec2.internal_4c30768f5a5da057efe9a1a98ac1fcde/haproxy/0.log" Jul 29 03:31:30.901025 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:30.900982 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c5488cb64-smstx_499fd96b-aced-4e44-95e1-5b81f206a24e/authorino/0.log" Jul 29 03:31:30.936134 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:30.936096 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-dc4c6_b7206433-81e2-4331-9650-33570a02e3e8/manager/0.log" Jul 29 03:31:30.967776 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:30.966628 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qqpwv_5a1ff45b-4a0b-441a-b112-411303f26976/manager/0.log" Jul 29 03:31:31.065490 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:31.065457 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4b9kw_790610a4-7db1-420f-b481-a41d41a828f0/registry-server/0.log" Jul 29 03:31:31.310429 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:31.306519 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pzkxr_63a5d434-45db-443a-9c54-97a01baab316/manager/0.log" Jul 29 03:31:33.488200 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:33.488166 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgsvz_743bdc7a-7233-4a46-b461-18c5883e949f/node-exporter/0.log" Jul 29 03:31:33.514931 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:33.514895 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgsvz_743bdc7a-7233-4a46-b461-18c5883e949f/kube-rbac-proxy/0.log" Jul 29 03:31:33.535387 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:33.535228 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgsvz_743bdc7a-7233-4a46-b461-18c5883e949f/init-textfile/0.log" Jul 29 03:31:35.422465 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.422430 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx"] Jul 29 03:31:35.435602 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.435567 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx"] Jul 29 03:31:35.435949 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.435930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.487952 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.487894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-lib-modules\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.487952 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.487948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-podres\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.488253 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.487982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-sys\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.488253 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.488063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-proc\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.488253 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.488118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2hz\" (UniqueName: \"kubernetes.io/projected/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-kube-api-access-2s2hz\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588691 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-lib-modules\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588691 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-podres\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-sys\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-proc\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2hz\" (UniqueName: \"kubernetes.io/projected/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-kube-api-access-2s2hz\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-sys\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-lib-modules\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-podres\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.588969 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.588918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-proc\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.596761 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.596734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2hz\" (UniqueName: \"kubernetes.io/projected/bbc23f1e-e177-448e-a0c4-cbc8c4f649cf-kube-api-access-2s2hz\") pod \"perf-node-gather-daemonset-6kgxx\" (UID: \"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf\") " pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.753171 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.753135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:35.937690 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:35.937114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx"] Jul 29 03:31:35.941737 ip-10-0-141-222 kubenswrapper[2579]: W0729 03:31:35.941692 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbbc23f1e_e177_448e_a0c4_cbc8c4f649cf.slice/crio-78ff8cc00a4305397d4c855cfb553112bb7ce2646ca4fc652591807c94714956 WatchSource:0}: Error finding container 78ff8cc00a4305397d4c855cfb553112bb7ce2646ca4fc652591807c94714956: Status 404 returned error can't find the container with id 78ff8cc00a4305397d4c855cfb553112bb7ce2646ca4fc652591807c94714956 Jul 29 03:31:36.623745 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:36.623703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" event={"ID":"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf","Type":"ContainerStarted","Data":"32286690aae7c1da72094a3916fc9e2c5d4b9f458278423a3e32abeaf74f4470"} Jul 29 03:31:36.624256 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:36.623777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" event={"ID":"bbc23f1e-e177-448e-a0c4-cbc8c4f649cf","Type":"ContainerStarted","Data":"78ff8cc00a4305397d4c855cfb553112bb7ce2646ca4fc652591807c94714956"} Jul 29 03:31:36.624771 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:36.624696 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:36.641003 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:36.640936 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" podStartSLOduration=1.640914967 podStartE2EDuration="1.640914967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 03:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 03:31:36.638962961 +0000 UTC m=+3386.699438362" watchObservedRunningTime="2026-07-29 03:31:36.640914967 +0000 UTC m=+3386.701390370" Jul 29 03:31:37.971127 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:37.971092 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gfgld_86640fb2-8e21-43a1-86be-31998eecabab/dns/0.log" Jul 29 03:31:37.990123 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:37.990091 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gfgld_86640fb2-8e21-43a1-86be-31998eecabab/kube-rbac-proxy/0.log" Jul 29 03:31:38.078777 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:38.078749 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6kpk_8d06fbb6-3aad-4a03-88dc-4a17c06ccaad/dns-node-resolver/0.log" Jul 29 03:31:38.633726 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:38.633676 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76c7df8f66-j5gz5_901124bb-feb2-4e60-aa16-749b7bfdb0e0/registry/0.log" Jul 29 03:31:38.657175 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:38.657145 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gm6b5_027117dd-7fa0-4382-ac51-1ebf12758b86/node-ca/0.log" Jul 29 03:31:39.603615 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:39.603578 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb_f3bd5a2c-bde0-4eb9-9308-c69a6eae8357/istio-proxy/0.log" Jul 29 03:31:39.798090 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:39.798057 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdc76d774-lmbzn_6ebef28c-7b32-4b43-a0de-4a197da67ef5/kube-auth-proxy/0.log" Jul 29 03:31:39.827888 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:39.827825 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k5qpd_379ad44b-8b7d-4957-a529-1be09e159bbc/istio-proxy/0.log" Jul 29 03:31:39.895962 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:39.895867 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k5qpd_379ad44b-8b7d-4957-a529-1be09e159bbc/istio-proxy/1.log" Jul 29 03:31:39.963424 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:39.963391 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dnxkj_7d17922f-85e0-4da5-96b0-3b7f52be73e3/payload-processing/0.log" Jul 29 03:31:40.818207 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:40.818170 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qlddw_ab1e2935-4cf7-42a8-9fc7-fc684d73378c/serve-healthcheck-canary/0.log" Jul 29 03:31:41.471937 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:41.471905 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lkqfc_f54510f7-6d8e-4430-948e-76c2c50a4dc7/kube-rbac-proxy/0.log" Jul 29 03:31:41.499921 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:41.499887 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lkqfc_f54510f7-6d8e-4430-948e-76c2c50a4dc7/exporter/0.log" Jul 29 03:31:41.524577 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:41.524544 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lkqfc_f54510f7-6d8e-4430-948e-76c2c50a4dc7/extractor/0.log" Jul 29 03:31:43.549330 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:43.549290 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7trrb_8f72f594-e7cc-4154-a777-77ba264f82f6/manager/0.log" Jul 29 03:31:43.650599 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:43.650561 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2j9bk/perf-node-gather-daemonset-6kgxx" Jul 29 03:31:43.981950 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:43.981909 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b7rjr_ebe6d1fa-9def-431b-8092-0ac85bdfe965/manager/0.log" Jul 29 03:31:44.259696 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:44.259623 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b54d569bf-pwmlg_81407d25-a23e-4c4f-8b86-59cc4bf3c539/manager/0.log" Jul 29 03:31:44.280194 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:44.280165 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-q6tnz_440221a6-cc0f-46a5-afe8-da54477362df/server/0.log" Jul 29 03:31:44.399367 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:44.399335 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2jldv_d8abdbca-5306-4318-acbc-bdbc6aef21a2/manager/0.log" Jul 29 03:31:44.424785 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:44.424755 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dbd45849d-857zf_4187b9e7-bba0-4ff9-a987-0179b32924e0/manager/0.log" Jul 29 03:31:44.444527 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:44.444502 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5dbd45849d-857zf_4187b9e7-bba0-4ff9-a987-0179b32924e0/copy-manifests/0.log" Jul 29 03:31:45.920635 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:45.920598 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb_f3bd5a2c-bde0-4eb9-9308-c69a6eae8357/istio-proxy/0.log" Jul 29 03:31:45.971270 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:45.971244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-twj6q_dfdd4671-9fcd-42c7-85f9-003da06edfb2/openshift-lws-operator/0.log" Jul 29 03:31:46.095782 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:46.095716 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdc76d774-lmbzn_6ebef28c-7b32-4b43-a0de-4a197da67ef5/kube-auth-proxy/0.log" Jul 29 03:31:46.119799 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:46.119764 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k5qpd_379ad44b-8b7d-4957-a529-1be09e159bbc/istio-proxy/0.log" Jul 29 03:31:46.188909 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:46.188859 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k5qpd_379ad44b-8b7d-4957-a529-1be09e159bbc/istio-proxy/1.log" Jul 29 03:31:46.241497 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:46.241383 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dnxkj_7d17922f-85e0-4da5-96b0-3b7f52be73e3/payload-processing/0.log" Jul 29 03:31:47.625518 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:47.625469 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cdf5tb_f3bd5a2c-bde0-4eb9-9308-c69a6eae8357/istio-proxy/0.log" Jul 29 03:31:47.795351 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:47.795322 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdc76d774-lmbzn_6ebef28c-7b32-4b43-a0de-4a197da67ef5/kube-auth-proxy/0.log" Jul 29 03:31:47.818535 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:47.818502 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k5qpd_379ad44b-8b7d-4957-a529-1be09e159bbc/istio-proxy/0.log" Jul 29 03:31:47.863973 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:47.863943 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k5qpd_379ad44b-8b7d-4957-a529-1be09e159bbc/istio-proxy/1.log" Jul 29 03:31:47.928531 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:47.928498 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dnxkj_7d17922f-85e0-4da5-96b0-3b7f52be73e3/payload-processing/0.log" Jul 29 03:31:51.107052 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:51.107000 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wt6rh_9e6388bc-0b0b-4183-ab8b-82a174d54462/migrator/0.log" Jul 29 03:31:51.125851 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:51.125822 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wt6rh_9e6388bc-0b0b-4183-ab8b-82a174d54462/graceful-termination/0.log" Jul 29 03:31:52.588346 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.588318 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jr8vc_21e67ffb-48ba-4975-99b6-3319df78688d/kube-multus-additional-cni-plugins/0.log" Jul 29 03:31:52.610136 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.610111 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jr8vc_21e67ffb-48ba-4975-99b6-3319df78688d/egress-router-binary-copy/0.log" Jul 29 03:31:52.630226 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.630183 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jr8vc_21e67ffb-48ba-4975-99b6-3319df78688d/cni-plugins/0.log" Jul 29 03:31:52.649647 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.649621 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jr8vc_21e67ffb-48ba-4975-99b6-3319df78688d/bond-cni-plugin/0.log" Jul 29 03:31:52.669158 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.669119 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jr8vc_21e67ffb-48ba-4975-99b6-3319df78688d/routeoverride-cni/0.log" Jul 29 03:31:52.691162 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.691133 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jr8vc_21e67ffb-48ba-4975-99b6-3319df78688d/whereabouts-cni-bincopy/0.log" Jul 29 03:31:52.710533 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.710500 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jr8vc_21e67ffb-48ba-4975-99b6-3319df78688d/whereabouts-cni/0.log" Jul 29 03:31:52.906067 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:52.905973 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-d6khs_2bbaf700-6f58-4abc-91ef-94e2e438103b/kube-multus/0.log" Jul 29 03:31:53.046119 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:53.046089 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bn4t5_bfbbffef-def3-42d4-b59d-0dee8edf10b0/network-metrics-daemon/0.log" Jul 29 03:31:53.064057 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:53.064032 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bn4t5_bfbbffef-def3-42d4-b59d-0dee8edf10b0/kube-rbac-proxy/0.log" Jul 29 03:31:54.124981 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.124938 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-controller/0.log" Jul 29 03:31:54.140186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.140117 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/1.log" Jul 29 03:31:54.140186 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.140166 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/ovn-acl-logging/0.log" Jul 29 03:31:54.157531 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.157489 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/kube-rbac-proxy-node/0.log" Jul 29 03:31:54.179206 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.179157 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/kube-rbac-proxy-ovn-metrics/0.log" Jul 29 03:31:54.198629 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.198595 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/northd/0.log" Jul 29 03:31:54.218212 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.218179 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/nbdb/0.log" Jul 29 03:31:54.240543 ip-10-0-141-222 kubenswrapper[2579]: I0729 03:31:54.240515 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwqp4_6f16b74f-467e-4774-af25-54144c4b3353/sbdb/0.log"