Sep 08 16:10:38.855236 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 08 16:10:38.855248 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 08 16:10:38.855256 ip-10-0-139-234 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 08 16:10:38.855545 ip-10-0-139-234 systemd[1]: Failed to start Kubernetes Kubelet. Sep 08 16:10:50.204892 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 08 16:10:50.204910 ip-10-0-139-234 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 042cf0913a3042cdb3873884c6808454 -- Sep 08 16:12:46.293565 ip-10-0-139-234 systemd[1]: Starting Kubernetes Kubelet... Sep 08 16:12:46.674753 ip-10-0-139-234 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:12:46.674753 ip-10-0-139-234 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 16:12:46.674753 ip-10-0-139-234 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:12:46.674753 ip-10-0-139-234 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 16:12:46.674753 ip-10-0-139-234 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:12:46.676167 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.676084 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 16:12:46.679839 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679819 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:12:46.679839 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679837 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:12:46.679839 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679841 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679844 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679848 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679851 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679854 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679857 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679860 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679864 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679869 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679873 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679876 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679879 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679882 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679885 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679888 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679892 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679894 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679898 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679901 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:12:46.679934 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679904 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679908 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679911 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679914 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679917 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679919 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679922 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679925 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679928 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679930 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679933 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679935 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679938 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679941 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679944 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679947 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679949 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679953 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679956 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679959 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:12:46.680419 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679961 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679964 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679966 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679969 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679971 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679974 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679976 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679979 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679981 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679985 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679988 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679991 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679993 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.679996 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680000 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680003 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680006 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680008 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680011 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680014 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:12:46.680903 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680016 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680019 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680022 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680024 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680027 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680030 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680032 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680035 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680037 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680040 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680043 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680046 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680049 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680052 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680055 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680057 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680060 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680063 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680065 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:12:46.681417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680068 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680070 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680073 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680075 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680077 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680080 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680082 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680087 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680091 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680096 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680099 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680103 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680106 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680109 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680112 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680115 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680117 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680120 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680123 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:12:46.681873 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680874 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680881 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680884 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680887 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680890 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680894 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680898 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680901 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680904 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680906 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680909 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680912 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680915 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680918 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680921 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680923 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680926 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680928 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680931 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680934 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:12:46.682382 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680938 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680943 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680947 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680952 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680956 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680960 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680964 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680968 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680972 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680979 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680985 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680989 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680994 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.680998 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681002 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681007 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681011 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681014 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681019 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:12:46.682881 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681024 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681028 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681031 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681035 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681040 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681044 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681048 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681052 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681058 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681061 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681065 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681070 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681074 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681080 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681083 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681086 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681089 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681091 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681094 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681097 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:12:46.683369 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681099 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681102 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681104 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681107 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681109 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681112 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681114 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681119 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681122 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681125 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681128 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681131 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681135 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681137 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681140 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681143 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681145 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681148 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681151 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:12:46.683861 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681154 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681156 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681159 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681161 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681164 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681167 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681170 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681173 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681176 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681178 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681181 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681184 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681186 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681189 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681191 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681194 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681196 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681199 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681201 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681204 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:12:46.684354 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681206 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681639 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681645 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681648 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681650 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681654 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681657 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681660 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681663 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681666 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681668 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681671 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681673 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681676 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681679 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681681 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681684 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681687 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681690 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681693 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:12:46.684852 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681696 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681698 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681702 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681704 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681707 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681710 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681713 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681715 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681718 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681720 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681723 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681725 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681729 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681733 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681736 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681739 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681743 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681746 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681749 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:12:46.685363 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681751 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681754 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681757 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681760 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681762 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681765 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681768 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681771 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681773 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681776 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681778 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681783 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681785 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681788 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681792 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681795 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681798 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681801 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681804 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:12:46.685831 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681807 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681810 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681813 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681815 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681818 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681821 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681824 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681827 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681830 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681832 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681835 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681837 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681840 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681851 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681854 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681856 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681859 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681862 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681865 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681868 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:12:46.686306 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681871 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681873 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681876 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681879 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681885 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681888 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681891 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681894 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681896 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681899 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681902 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681904 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681907 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681910 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681912 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681915 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681918 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681920 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681923 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681925 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:12:46.686792 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681928 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.681930 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682010 2555 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682053 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682061 2555 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682066 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682071 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682074 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682079 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682083 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682087 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682090 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682093 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682097 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682100 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682103 2555 flags.go:64] FLAG: --cgroup-root="" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682109 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682112 2555 flags.go:64] FLAG: --client-ca-file="" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682116 2555 flags.go:64] FLAG: --cloud-provider="external" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682119 2555 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682123 2555 flags.go:64] FLAG: --cluster-domain="" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682126 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682129 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682132 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 16:12:46.687356 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682136 2555 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682140 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682143 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682146 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682149 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682153 2555 flags.go:64] FLAG: --contention-profiling="false" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682155 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682159 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682162 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682165 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682169 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682172 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682175 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682178 2555 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682181 2555 flags.go:64] FLAG: --enable-server="true" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682187 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682192 2555 flags.go:64] FLAG: --event-burst="100" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682195 2555 flags.go:64] FLAG: --event-qps="50" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682198 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682201 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682204 2555 flags.go:64] FLAG: --eviction-hard="" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682208 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682211 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682214 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682219 2555 flags.go:64] FLAG: --eviction-soft="" Sep 08 16:12:46.687940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682222 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682225 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682228 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682231 2555 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682234 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682237 2555 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682254 2555 flags.go:64] FLAG: --feature-gates="" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682258 2555 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682262 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682265 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682269 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682272 2555 flags.go:64] FLAG: --healthz-port="10248" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682276 2555 flags.go:64] FLAG: --help="false" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682279 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-139-234.ec2.internal" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682282 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682285 2555 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682288 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682292 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682295 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682298 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682301 2555 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682304 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682307 2555 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 16:12:46.688570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682311 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682314 2555 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682317 2555 flags.go:64] FLAG: --kube-reserved="" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682320 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682323 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682326 2555 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682329 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682332 2555 flags.go:64] FLAG: --lock-file="" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682337 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682341 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682344 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682349 2555 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682352 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682355 2555 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682359 2555 flags.go:64] FLAG: --logging-format="text" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682362 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682365 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682368 2555 flags.go:64] FLAG: --manifest-url="" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682371 2555 flags.go:64] FLAG: --manifest-url-header="" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682376 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682379 2555 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682383 2555 flags.go:64] FLAG: --max-pods="110" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682386 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682390 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682393 2555 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 16:12:46.689151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682396 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682399 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682402 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682405 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682439 2555 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682443 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682446 2555 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682450 2555 flags.go:64] FLAG: --pod-cidr="" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682454 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682461 2555 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682464 2555 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682467 2555 flags.go:64] FLAG: --pods-per-core="0" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682470 2555 flags.go:64] FLAG: --port="10250" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682474 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682477 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0295fc14b747c52bf" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682481 2555 flags.go:64] FLAG: --qos-reserved="" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682485 2555 flags.go:64] FLAG: --read-only-port="10255" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682489 2555 flags.go:64] FLAG: --register-node="true" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682492 2555 flags.go:64] FLAG: --register-with-taints="" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682496 2555 flags.go:64] FLAG: --registry-burst="10" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682499 2555 flags.go:64] FLAG: --registry-qps="5" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682502 2555 flags.go:64] FLAG: --reserved-cpus="" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682504 2555 flags.go:64] FLAG: --reserved-memory="" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682509 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682512 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 16:12:46.689805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682515 2555 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682518 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682521 2555 flags.go:64] FLAG: --runonce="false" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682524 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682527 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682530 2555 flags.go:64] FLAG: --seccomp-default="false" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682533 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682535 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682539 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682542 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682545 2555 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682548 2555 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682551 2555 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682554 2555 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682557 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682560 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682563 2555 flags.go:64] FLAG: --system-cgroups="" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682566 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682572 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682575 2555 flags.go:64] FLAG: --tls-cert-file="" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682579 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682583 2555 flags.go:64] FLAG: --tls-min-version="" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682586 2555 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682590 2555 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682593 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 16:12:46.690440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682596 2555 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682599 2555 flags.go:64] FLAG: --v="2" Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682603 2555 flags.go:64] FLAG: --version="false" Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682607 2555 flags.go:64] FLAG: --vmodule="" Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682611 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.682615 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682740 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682744 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682748 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682751 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682754 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682757 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682760 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682762 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682767 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682771 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682774 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682776 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682779 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682783 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682785 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:12:46.691108 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682788 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682790 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682793 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682796 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682799 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682801 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682805 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682808 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682811 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682815 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682818 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682821 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682823 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682826 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682829 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682831 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682834 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682837 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682839 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682842 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:12:46.691633 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682844 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682847 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682850 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682852 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682855 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682858 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682860 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682863 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682865 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682868 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682870 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682873 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682876 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682879 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682882 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682884 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682887 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682890 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682895 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682898 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:12:46.692124 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682900 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682904 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682907 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682910 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682912 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682915 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682918 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682921 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682923 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682926 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682928 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682931 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682935 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682938 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682941 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682944 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682947 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682949 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682952 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:12:46.692717 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682955 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682957 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682960 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682962 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682965 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682968 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682970 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682974 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682978 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682980 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682984 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682988 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682990 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682993 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.682997 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683000 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683003 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683005 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683008 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:12:46.693183 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683010 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683014 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683016 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683019 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683021 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.683024 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.683742 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.691299 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.691317 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.692098 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.692109 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692187 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692192 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692195 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692198 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:12:46.693670 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692201 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692204 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692209 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692213 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692216 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692219 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692222 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692225 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692228 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692231 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692234 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692236 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692239 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692256 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692259 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692262 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692264 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692267 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692270 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692273 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:12:46.694049 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692275 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692277 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692280 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692283 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692286 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692295 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692298 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692301 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692304 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692306 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692309 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692311 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692314 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692316 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692319 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692322 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692324 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692327 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692331 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:12:46.694570 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692336 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692339 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692342 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692345 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692348 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692351 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692354 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692357 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692359 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692362 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692364 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692367 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692370 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692372 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692374 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692377 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692379 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692382 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692384 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692392 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:12:46.695027 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692395 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692397 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692400 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692402 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692405 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692407 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692410 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692412 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692415 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692417 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692420 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692422 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692425 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692427 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692430 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692433 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692435 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692438 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692440 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692443 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:12:46.695579 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692446 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692449 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692451 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692453 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692456 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692459 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692461 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692464 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692467 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692469 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692472 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692474 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692483 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692486 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692489 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:12:46.696060 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692491 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.692497 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692653 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692658 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692661 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692664 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692667 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692669 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692672 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692674 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692677 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692680 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692682 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692685 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692688 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692690 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:12:46.696509 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692693 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692696 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692699 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692701 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692704 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692707 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692712 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692715 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692717 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692720 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692723 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692726 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692728 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692731 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692741 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692744 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692747 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692750 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692752 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:12:46.696907 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692755 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692759 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692762 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692765 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692769 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692772 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692775 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692778 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692781 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692784 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692786 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692789 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692792 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692796 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692798 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692801 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692804 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692807 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692810 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:12:46.697426 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692813 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692816 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692818 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692821 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692824 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692826 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692829 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692831 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692834 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692842 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692845 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692847 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692850 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692852 2555 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692855 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692857 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692860 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692862 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692865 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:12:46.697896 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692868 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692870 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692873 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692876 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692879 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692881 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692884 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692888 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692891 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692894 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692896 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692900 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692902 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692905 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692907 2555 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692910 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692912 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692915 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692917 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692920 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:12:46.698377 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692923 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692926 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692928 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692936 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692939 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692942 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692945 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:46.692947 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.692952 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.693103 2555 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.696914 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.697857 2555 server.go:1019] "Starting client certificate rotation" Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.697953 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:12:46.698877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.697991 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:12:46.701769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.701743 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:12:46.703097 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.703074 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:12:46.715898 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.715877 2555 log.go:25] "Validated CRI v1 runtime API" Sep 08 16:12:46.719543 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.719525 2555 log.go:25] "Validated CRI v1 image API" Sep 08 16:12:46.720458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.720440 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 16:12:46.725342 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.725319 2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b9258430-873b-4444-b529-88c14bb8b714:/dev/nvme0n1p3 caddc4d2-c056-48d3-938a-0dfab09412d4:/dev/nvme0n1p4] Sep 08 16:12:46.725453 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.725341 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 16:12:46.731108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.730974 2555 manager.go:216] Machine: {Timestamp:2026-09-08 16:12:46.729614489 +0000 UTC m=+0.324613239 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3130956 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee866184954d440586862f8bb2eaf SystemUUID:ec2ee866-1849-54d4-4058-6862f8bb2eaf BootID:042cf091-3a30-42cd-b387-3884c6808454 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:37:d1:59:8d:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:37:d1:59:8d:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:d4:a7:80:d8:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 16:12:46.731108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.731091 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 16:12:46.731336 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.731279 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 16:12:46.731793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.731773 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:12:46.732489 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.732458 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 16:12:46.732680 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.732496 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 16:12:46.733199 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.733187 2555 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 16:12:46.733268 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.733204 2555 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 16:12:46.733268 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.733215 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:12:46.733955 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.733943 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:12:46.735389 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.735377 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 16:12:46.738875 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.738859 2555 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:12:46.739025 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739014 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 16:12:46.739094 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739054 2555 kubelet.go:486] "Attempting to sync node with API server" Sep 08 16:12:46.739094 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739069 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 16:12:46.739222 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739099 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 16:12:46.739222 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739115 2555 kubelet.go:398] "Adding apiserver pod source" Sep 08 16:12:46.739222 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739128 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 16:12:46.739585 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739572 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:12:46.739650 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.739594 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:12:46.743088 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.743070 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 16:12:46.745399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.745384 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 16:12:46.745474 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.745415 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 16:12:46.746108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746095 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 16:12:46.746144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746115 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 16:12:46.746144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746121 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 16:12:46.746144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746130 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 16:12:46.746144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746141 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 16:12:46.746277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746148 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 16:12:46.746277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746162 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 16:12:46.746277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746167 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 16:12:46.746277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746173 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 16:12:46.746277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746179 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 16:12:46.746277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746192 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 16:12:46.746277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746200 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 16:12:46.746932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746922 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 16:12:46.746932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.746932 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 16:12:46.751646 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.751629 2555 server.go:1268] "Started kubelet" Sep 08 16:12:46.751820 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.751775 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 16:12:46.751937 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.751885 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 16:12:46.752041 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.751954 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 16:12:46.752097 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.752041 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 16:12:46.752181 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.752159 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 16:12:46.752234 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.752182 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 16:12:46.753261 ip-10-0-139-234 systemd[1]: Started Kubernetes Kubelet. Sep 08 16:12:46.753566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.753552 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 16:12:46.754691 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.754672 2555 server.go:310] "Adding debug handlers to kubelet server" Sep 08 16:12:46.757035 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.757015 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 16:12:46.758034 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758017 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 16:12:46.758126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758056 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 16:12:46.758126 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.758096 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:46.758625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758607 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 16:12:46.758753 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758610 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 16:12:46.758884 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758866 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 16:12:46.758962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758888 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 16:12:46.758962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758901 2555 factory.go:55] Registering systemd factory Sep 08 16:12:46.758962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.758910 2555 factory.go:223] Registration of the systemd container factory successfully Sep 08 16:12:46.759120 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.759006 2555 reconstruct.go:97] "Volume reconstruction finished" Sep 08 16:12:46.759120 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.759014 2555 reconciler.go:29] "Reconciler: start to sync state" Sep 08 16:12:46.759205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.759140 2555 factory.go:153] Registering CRI-O factory Sep 08 16:12:46.759205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.759153 2555 factory.go:223] Registration of the crio container factory successfully Sep 08 16:12:46.759205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.759166 2555 factory.go:103] Registering Raw factory Sep 08 16:12:46.759205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.759190 2555 manager.go:1196] Started watching for new ooms in manager Sep 08 16:12:46.759570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.759557 2555 manager.go:349] Starting recovery of all containers Sep 08 16:12:46.759940 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.758983 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-234.ec2.internal.18d3647d669ec8b8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.ec2.internal,UID:ip-10-0-139-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.751590584 +0000 UTC m=+0.346589333,LastTimestamp:2026-09-08 16:12:46.751590584 +0000 UTC m=+0.346589333,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.ec2.internal,}" Sep 08 16:12:46.767236 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.767207 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 16:12:46.767347 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.767265 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 16:12:46.772151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.772131 2555 manager.go:354] Recovery completed Sep 08 16:12:46.779257 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.779229 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:12:46.781617 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.781591 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:12:46.781706 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.781638 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:12:46.781706 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.781656 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:12:46.782365 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.782350 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 16:12:46.782365 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.782364 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 16:12:46.782475 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.782371 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:12:46.782475 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.782381 2555 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:12:46.783139 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.783124 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:12:46.783205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.783148 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:12:46.783205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.783158 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:12:46.783653 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.783638 2555 policy_none.go:49] "None policy: Start" Sep 08 16:12:46.783711 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.783656 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 16:12:46.783711 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.783667 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 16:12:46.783968 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.783862 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-234.ec2.internal.18d3647d6868f4d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.ec2.internal,UID:ip-10-0-139-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.781617366 +0000 UTC m=+0.376616121,LastTimestamp:2026-09-08 16:12:46.781617366 +0000 UTC m=+0.376616121,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.ec2.internal,}" Sep 08 16:12:46.784969 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.784956 2555 policy_none.go:47] "Start" Sep 08 16:12:46.793804 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.793777 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dtdqb" Sep 08 16:12:46.795206 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.795112 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-234.ec2.internal.18d3647d68696367 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.ec2.internal,UID:ip-10-0-139-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-234.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.781645671 +0000 UTC m=+0.376644422,LastTimestamp:2026-09-08 16:12:46.781645671 +0000 UTC m=+0.376644422,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.ec2.internal,}" Sep 08 16:12:46.806218 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.806191 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dtdqb" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.808622 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-234.ec2.internal.18d3647d6869a116 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.ec2.internal,UID:ip-10-0-139-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-234.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.781661462 +0000 UTC m=+0.376660216,LastTimestamp:2026-09-08 16:12:46.781661462 +0000 UTC m=+0.376660216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.ec2.internal,}" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.810923 2555 manager.go:337] "Starting Device Plugin manager" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.810968 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.810981 2555 server.go:87] "Starting device plugin registration server" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.811223 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.811234 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.811350 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.811434 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.811442 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.812013 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 16:12:46.821333 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.812055 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:46.911785 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.911739 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:12:46.912880 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.912861 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:12:46.912996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.912890 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:12:46.912996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.912904 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:12:46.912996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.912930 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-234.ec2.internal" Sep 08 16:12:46.922827 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.922807 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-234.ec2.internal" Sep 08 16:12:46.922894 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.922837 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-234.ec2.internal\": node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:46.928742 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.928654 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 16:12:46.930092 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.930070 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 16:12:46.930194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.930123 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 16:12:46.930194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.930150 2555 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 16:12:46.930307 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.930195 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 16:12:46.934493 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:46.934472 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:12:46.953001 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:46.952980 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.031877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.031236 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal"] Sep 08 16:12:47.031982 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.031929 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:12:47.033002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.032986 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:12:47.033091 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.033018 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:12:47.033091 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.033028 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:12:47.034494 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.034481 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:12:47.034613 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.034598 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.034704 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.034646 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:12:47.035299 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.035283 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:12:47.035371 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.035310 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:12:47.035371 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.035319 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:12:47.035371 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.035284 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:12:47.035504 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.035393 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:12:47.035504 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.035410 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:12:47.036526 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.036511 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.036607 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.036543 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:12:47.038111 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.038092 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:12:47.038205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.038123 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:12:47.038205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.038133 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:12:47.053484 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.053460 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.060818 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.060786 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b512f4dee5198201752a4ab64dc951d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal\" (UID: \"7b512f4dee5198201752a4ab64dc951d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.060901 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.060825 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b512f4dee5198201752a4ab64dc951d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal\" (UID: \"7b512f4dee5198201752a4ab64dc951d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.060901 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.060844 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bcaf88a8105917bc3decd15ff63fd729-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.ec2.internal\" (UID: \"bcaf88a8105917bc3decd15ff63fd729\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.066054 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.066034 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.ec2.internal\" not found" node="ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.070676 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.070656 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.ec2.internal\" not found" node="ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.154395 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.154360 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.161475 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.161451 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b512f4dee5198201752a4ab64dc951d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal\" (UID: \"7b512f4dee5198201752a4ab64dc951d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.161553 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.161482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b512f4dee5198201752a4ab64dc951d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal\" (UID: \"7b512f4dee5198201752a4ab64dc951d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.161553 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.161502 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bcaf88a8105917bc3decd15ff63fd729-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.ec2.internal\" (UID: \"bcaf88a8105917bc3decd15ff63fd729\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.161616 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.161573 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b512f4dee5198201752a4ab64dc951d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal\" (UID: \"7b512f4dee5198201752a4ab64dc951d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.161616 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.161583 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bcaf88a8105917bc3decd15ff63fd729-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.ec2.internal\" (UID: \"bcaf88a8105917bc3decd15ff63fd729\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.161675 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.161573 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b512f4dee5198201752a4ab64dc951d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal\" (UID: \"7b512f4dee5198201752a4ab64dc951d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.254564 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.254481 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.356855 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.356823 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.369302 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.369276 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.374075 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.373755 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.459347 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.459294 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.565777 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.565707 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.600996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.600968 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:12:47.666586 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:47.666545 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-234.ec2.internal\" not found" Sep 08 16:12:47.698414 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.698364 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 16:12:47.698889 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.698552 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:12:47.698889 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.698574 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:12:47.761018 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.760989 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:12:47.780129 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.780097 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:12:47.809128 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.809063 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:07:47 +0000 UTC" deadline="2028-04-06 01:49:25.536501074 +0000 UTC" Sep 08 16:12:47.809128 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.809118 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13809h36m37.727387072s" Sep 08 16:12:47.859957 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.859870 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.874582 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.874555 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 16:12:47.875754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.875740 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" Sep 08 16:12:47.885558 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:47.885537 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 16:12:48.236278 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:48.236224 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcaf88a8105917bc3decd15ff63fd729.slice/crio-1a562a66372d2ffe3771abf4cfca53863465f96b76bde891dc78fdebf2b8d79f WatchSource:0}: Error finding container 1a562a66372d2ffe3771abf4cfca53863465f96b76bde891dc78fdebf2b8d79f: Status 404 returned error can't find the container with id 1a562a66372d2ffe3771abf4cfca53863465f96b76bde891dc78fdebf2b8d79f Sep 08 16:12:48.241876 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.241858 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:12:48.580336 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:48.580291 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b512f4dee5198201752a4ab64dc951d.slice/crio-b27cfd68b7ec4ecc9f797dc3bd1e20a611f17150ddd271c4318cadc88664f0b9 WatchSource:0}: Error finding container b27cfd68b7ec4ecc9f797dc3bd1e20a611f17150ddd271c4318cadc88664f0b9: Status 404 returned error can't find the container with id b27cfd68b7ec4ecc9f797dc3bd1e20a611f17150ddd271c4318cadc88664f0b9 Sep 08 16:12:48.742953 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.742907 2555 apiserver.go:52] "Watching apiserver" Sep 08 16:12:48.750507 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.750476 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 16:12:48.750872 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.750851 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-68k9z","openshift-ovn-kubernetes/ovnkube-node-smsxf","kube-system/konnectivity-agent-tqbd2","kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5","openshift-cluster-node-tuning-operator/tuned-xhlvb","openshift-multus/multus-additional-cni-plugins-lj8pc","openshift-network-operator/iptables-alerter-lnzlp","kube-system/global-pull-secret-syncer-xtsbg","openshift-dns/node-resolver-bxdnw","openshift-image-registry/node-ca-nsbsw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal","openshift-multus/multus-hz52h","openshift-multus/network-metrics-daemon-fbd6c"] Sep 08 16:12:48.752580 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.752562 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.753867 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.753844 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.754854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.754835 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 16:12:48.754970 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.754865 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 16:12:48.754970 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.754913 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6ksrb\"" Sep 08 16:12:48.755163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.755146 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:48.756295 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.756275 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 16:12:48.756479 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.756461 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:48.756568 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.756546 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:12:48.756635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.756557 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 16:12:48.756635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.756614 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nznrf\"" Sep 08 16:12:48.756736 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.756685 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 16:12:48.756789 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.756779 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 16:12:48.757632 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.757608 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 16:12:48.757632 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.757622 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4lwn4\"" Sep 08 16:12:48.757778 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.757649 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 16:12:48.758516 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.758499 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 16:12:48.758631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.758610 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.759972 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.759953 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.760760 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.760740 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 16:12:48.760829 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.760814 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 16:12:48.760984 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.760968 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 16:12:48.761078 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.761056 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l4g28\"" Sep 08 16:12:48.761251 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.761231 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.761391 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.761322 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:12:48.762026 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.761985 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 16:12:48.762273 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.762251 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-95v2w\"" Sep 08 16:12:48.762365 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.762290 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:12:48.762932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.762913 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.764493 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.764473 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.764939 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.764919 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 16:12:48.765179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.765162 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 16:12:48.765258 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.765170 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zt44w\"" Sep 08 16:12:48.766029 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.765994 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:48.766505 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.766478 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:12:48.766828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.766805 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 16:12:48.768736 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.768714 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 16:12:48.768968 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.768950 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 16:12:48.769615 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.769590 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9vp6\"" Sep 08 16:12:48.769615 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.769609 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 16:12:48.770558 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.769976 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 16:12:48.770558 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.770107 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.770558 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.770482 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 16:12:48.771256 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.771236 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:12:48.772314 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772247 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-netns\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.772314 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772298 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:48.772314 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772326 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-sys-fs\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772350 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysconfig\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772373 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-system-cni-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772376 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b88bx\"" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772400 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-socket-dir-parent\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772376 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772428 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-cni-multus\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772453 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-socket-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772478 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtjm\" (UniqueName: \"kubernetes.io/projected/a9142923-ec0a-459f-ab44-3594ebfa1c86-kube-api-access-fvtjm\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772505 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0ff108a-6700-47c6-b566-f6e6740810e1-hosts-file\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772516 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 16:12:48.772539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e64c5b01-186f-4002-827e-e2821aad92b0-agent-certs\") pod \"konnectivity-agent-tqbd2\" (UID: \"e64c5b01-186f-4002-827e-e2821aad92b0\") " pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772545 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772565 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772584 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-modprobe-d\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772624 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772637 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-sys\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772687 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff108a-6700-47c6-b566-f6e6740810e1-tmp-dir\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772712 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-cnibin\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772742 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-hostroot\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-conf-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772781 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdsz\" (UniqueName: \"kubernetes.io/projected/a42de246-aa61-4934-96e1-07e4fb87f898-kube-api-access-6qdsz\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-device-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772834 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-kubernetes\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772857 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-k8s-cni-cncf-io\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772878 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a42de246-aa61-4934-96e1-07e4fb87f898-multus-daemon-config\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772897 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-etc-kubernetes\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772922 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-etc-selinux\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772947 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysctl-d\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772960 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-run\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772975 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-tuned\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.772996 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-cni-bin\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773036 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e64c5b01-186f-4002-827e-e2821aad92b0-konnectivity-ca\") pod \"konnectivity-agent-tqbd2\" (UID: \"e64c5b01-186f-4002-827e-e2821aad92b0\") " pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773057 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-registration-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773075 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-lib-modules\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773090 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-var-lib-kubelet\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773123 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-cni-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773158 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-kubelet\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773195 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsr9d\" (UniqueName: \"kubernetes.io/projected/3fb613c0-0100-414c-b958-e753b2d9a521-kube-api-access-fsr9d\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773217 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-systemd\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773233 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-host\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773270 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66p9f\" (UniqueName: \"kubernetes.io/projected/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-kube-api-access-66p9f\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773304 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-multus-certs\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773338 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysctl-conf\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773381 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-tmp\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773424 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phdsq\" (UniqueName: \"kubernetes.io/projected/f0ff108a-6700-47c6-b566-f6e6740810e1-kube-api-access-phdsq\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.773735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.773458 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a42de246-aa61-4934-96e1-07e4fb87f898-cni-binary-copy\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.774647 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.774628 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:12:48.774709 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.774632 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 16:12:48.775357 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.775338 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x7mxl\"" Sep 08 16:12:48.775648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.775632 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 16:12:48.795705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.795683 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r5jzs" Sep 08 16:12:48.809759 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.809736 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r5jzs" Sep 08 16:12:48.860339 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.860246 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 16:12:48.873931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.873897 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:48.873931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.873936 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff108a-6700-47c6-b566-f6e6740810e1-tmp-dir\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.873957 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-conf-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874008 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-device-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874038 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-kubernetes\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66p9f\" (UniqueName: \"kubernetes.io/projected/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-kube-api-access-66p9f\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874087 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-conf-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874090 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a42de246-aa61-4934-96e1-07e4fb87f898-multus-daemon-config\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874113 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-device-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874137 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysctl-d\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874160 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-slash\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.874180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-cni-bin\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874198 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-lib-modules\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874217 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-kubernetes\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874224 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-var-lib-kubelet\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874277 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-system-cni-dir\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874287 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-var-lib-kubelet\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874294 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-cni-bin\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874302 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysctl-d\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874318 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovnkube-script-lib\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874325 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-lib-modules\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874351 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsr9d\" (UniqueName: \"kubernetes.io/projected/3fb613c0-0100-414c-b958-e753b2d9a521-kube-api-access-fsr9d\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874389 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-host\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874413 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4234fc50-378f-4ac5-a183-0cb15a2365a5-kubelet-config\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874435 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-run-netns\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874451 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02929572-64d2-492c-8602-3a3dc9bc9701-host\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874461 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-host\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874488 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysctl-conf\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.874682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874510 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-tmp\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874526 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874575 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-var-lib-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874614 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a42de246-aa61-4934-96e1-07e4fb87f898-cni-binary-copy\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874659 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-netns\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-sys-fs\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874701 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysctl-conf\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874754 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-netns\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874756 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a42de246-aa61-4934-96e1-07e4fb87f898-multus-daemon-config\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874782 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-cni-bin\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874759 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-sys-fs\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874809 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-system-cni-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874853 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-system-cni-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874871 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0ff108a-6700-47c6-b566-f6e6740810e1-tmp-dir\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874908 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-socket-dir-parent\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874930 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.874938 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-socket-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875007 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-socket-dir-parent\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875011 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtjm\" (UniqueName: \"kubernetes.io/projected/a9142923-ec0a-459f-ab44-3594ebfa1c86-kube-api-access-fvtjm\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875032 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-socket-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875057 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-cnibin\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875083 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875100 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-systemd\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875106 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a42de246-aa61-4934-96e1-07e4fb87f898-cni-binary-copy\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875120 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8ll4\" (UniqueName: \"kubernetes.io/projected/02929572-64d2-492c-8602-3a3dc9bc9701-kube-api-access-z8ll4\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875170 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0ff108a-6700-47c6-b566-f6e6740810e1-hosts-file\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875211 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875229 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-sys\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875270 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0ff108a-6700-47c6-b566-f6e6740810e1-hosts-file\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875275 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875313 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-iptables-alerter-script\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875335 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-sys\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875363 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-etc-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875380 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875417 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-cnibin\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.875993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875441 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-hostroot\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875453 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-cnibin\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875464 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdsz\" (UniqueName: \"kubernetes.io/projected/a42de246-aa61-4934-96e1-07e4fb87f898-kube-api-access-6qdsz\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875489 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-hostroot\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875543 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-run\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875657 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-tuned\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875655 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-run\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875705 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-k8s-cni-cncf-io\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875742 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-etc-kubernetes\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-etc-selinux\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875806 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzs7s\" (UniqueName: \"kubernetes.io/projected/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-kube-api-access-qzs7s\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875830 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875835 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-etc-kubernetes\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875871 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-node-log\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875875 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-k8s-cni-cncf-io\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875900 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-log-socket\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875923 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-etc-selinux\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875950 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02929572-64d2-492c-8602-3a3dc9bc9701-serviceca\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.876786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e64c5b01-186f-4002-827e-e2821aad92b0-konnectivity-ca\") pod \"konnectivity-agent-tqbd2\" (UID: \"e64c5b01-186f-4002-827e-e2821aad92b0\") " pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.875986 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-registration-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876012 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-run-ovn-kubernetes\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876043 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a9142923-ec0a-459f-ab44-3594ebfa1c86-registration-dir\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876065 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-cni-netd\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876089 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovn-node-metrics-cert\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876114 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-cni-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876139 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-kubelet\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876164 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-systemd\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876223 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-kubelet\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876256 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-multus-cni-dir\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876273 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876301 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-ovn\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876321 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-systemd\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876330 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-multus-certs\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876360 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-host-slash\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.877762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876411 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-run-multus-certs\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876454 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phdsq\" (UniqueName: \"kubernetes.io/projected/f0ff108a-6700-47c6-b566-f6e6740810e1-kube-api-access-phdsq\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876481 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e64c5b01-186f-4002-827e-e2821aad92b0-konnectivity-ca\") pod \"konnectivity-agent-tqbd2\" (UID: \"e64c5b01-186f-4002-827e-e2821aad92b0\") " pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876585 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876604 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysconfig\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876644 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646d2\" (UniqueName: \"kubernetes.io/projected/091076f1-c7ed-4f20-8191-2fbf472b2a92-kube-api-access-646d2\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876682 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-cni-multus\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876689 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-sysconfig\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876725 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a42de246-aa61-4934-96e1-07e4fb87f898-host-var-lib-cni-multus\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876725 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4234fc50-378f-4ac5-a183-0cb15a2365a5-dbus\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.876746 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876758 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-kubelet\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876785 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-env-overrides\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.876815 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:49.376797286 +0000 UTC m=+2.971796029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876844 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e64c5b01-186f-4002-827e-e2821aad92b0-agent-certs\") pod \"konnectivity-agent-tqbd2\" (UID: \"e64c5b01-186f-4002-827e-e2821aad92b0\") " pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876864 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-modprobe-d\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.878682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876887 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8tgb\" (UniqueName: \"kubernetes.io/projected/266f7a5b-0582-4cc3-945f-0229e086fad3-kube-api-access-z8tgb\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.879250 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876912 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-systemd-units\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.879250 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876952 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovnkube-config\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.879250 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.876959 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-modprobe-d\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.879250 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.878886 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-tmp\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.879250 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.878949 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-etc-tuned\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.879250 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.879235 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e64c5b01-186f-4002-827e-e2821aad92b0-agent-certs\") pod \"konnectivity-agent-tqbd2\" (UID: \"e64c5b01-186f-4002-827e-e2821aad92b0\") " pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:48.882680 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.882650 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66p9f\" (UniqueName: \"kubernetes.io/projected/61544e0b-52f3-4842-8d62-ffcc0e2df2f1-kube-api-access-66p9f\") pod \"tuned-xhlvb\" (UID: \"61544e0b-52f3-4842-8d62-ffcc0e2df2f1\") " pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:48.883355 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.883299 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsr9d\" (UniqueName: \"kubernetes.io/projected/3fb613c0-0100-414c-b958-e753b2d9a521-kube-api-access-fsr9d\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:48.883523 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.883506 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdsz\" (UniqueName: \"kubernetes.io/projected/a42de246-aa61-4934-96e1-07e4fb87f898-kube-api-access-6qdsz\") pod \"multus-hz52h\" (UID: \"a42de246-aa61-4934-96e1-07e4fb87f898\") " pod="openshift-multus/multus-hz52h" Sep 08 16:12:48.883731 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.883714 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtjm\" (UniqueName: \"kubernetes.io/projected/a9142923-ec0a-459f-ab44-3594ebfa1c86-kube-api-access-fvtjm\") pod \"aws-ebs-csi-driver-node-6c2h5\" (UID: \"a9142923-ec0a-459f-ab44-3594ebfa1c86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:48.886715 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.886688 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phdsq\" (UniqueName: \"kubernetes.io/projected/f0ff108a-6700-47c6-b566-f6e6740810e1-kube-api-access-phdsq\") pod \"node-resolver-bxdnw\" (UID: \"f0ff108a-6700-47c6-b566-f6e6740810e1\") " pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:48.934577 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.934517 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" event={"ID":"7b512f4dee5198201752a4ab64dc951d","Type":"ContainerStarted","Data":"b27cfd68b7ec4ecc9f797dc3bd1e20a611f17150ddd271c4318cadc88664f0b9"} Sep 08 16:12:48.935758 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.935734 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" event={"ID":"bcaf88a8105917bc3decd15ff63fd729","Type":"ContainerStarted","Data":"1a562a66372d2ffe3771abf4cfca53863465f96b76bde891dc78fdebf2b8d79f"} Sep 08 16:12:48.978088 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978048 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978088 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978086 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-run-ovn-kubernetes\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978103 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-cni-netd\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978121 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovn-node-metrics-cert\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978153 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978177 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-ovn\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978203 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-cni-netd\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978229 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-host-slash\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978183 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-run-ovn-kubernetes\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978294 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-host-slash\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978321 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646d2\" (UniqueName: \"kubernetes.io/projected/091076f1-c7ed-4f20-8191-2fbf472b2a92-kube-api-access-646d2\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978348 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4234fc50-378f-4ac5-a183-0cb15a2365a5-dbus\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978374 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-kubelet\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.978399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978400 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-env-overrides\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978419 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4234fc50-378f-4ac5-a183-0cb15a2365a5-dbus\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8tgb\" (UniqueName: \"kubernetes.io/projected/266f7a5b-0582-4cc3-945f-0229e086fad3-kube-api-access-z8tgb\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.978434 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978458 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-systemd-units\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978466 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-kubelet\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978482 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovnkube-config\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.978500 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret podName:4234fc50-378f-4ac5-a183-0cb15a2365a5 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:49.478480504 +0000 UTC m=+3.073479262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret") pod "global-pull-secret-syncer-xtsbg" (UID: "4234fc50-378f-4ac5-a183-0cb15a2365a5") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978622 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-ovn\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978634 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978678 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-slash\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978714 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-system-cni-dir\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978681 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-systemd-units\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978740 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovnkube-script-lib\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978772 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4234fc50-378f-4ac5-a183-0cb15a2365a5-kubelet-config\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978782 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-slash\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978798 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-run-netns\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978825 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02929572-64d2-492c-8602-3a3dc9bc9701-host\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978854 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978882 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-var-lib-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978902 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-system-cni-dir\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978952 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-cni-bin\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978959 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4234fc50-378f-4ac5-a183-0cb15a2365a5-kubelet-config\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978909 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-cni-bin\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.978998 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-run-netns\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979004 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-cnibin\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979017 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979031 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979037 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02929572-64d2-492c-8602-3a3dc9bc9701-host\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979055 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-env-overrides\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-systemd\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979081 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-var-lib-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979096 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8ll4\" (UniqueName: \"kubernetes.io/projected/02929572-64d2-492c-8602-3a3dc9bc9701-kube-api-access-z8ll4\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979124 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-cnibin\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.979986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979129 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-iptables-alerter-script\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979142 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovnkube-config\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979161 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-etc-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979198 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979204 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-run-systemd\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979232 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/266f7a5b-0582-4cc3-945f-0229e086fad3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979249 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs7s\" (UniqueName: \"kubernetes.io/projected/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-kube-api-access-qzs7s\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-etc-openvswitch\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979283 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979334 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-node-log\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979339 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979360 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-log-socket\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979386 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02929572-64d2-492c-8602-3a3dc9bc9701-serviceca\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979392 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovnkube-script-lib\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979450 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-node-log\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979497 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/091076f1-c7ed-4f20-8191-2fbf472b2a92-log-socket\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979699 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.980525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979729 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-iptables-alerter-script\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:48.981095 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979788 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/266f7a5b-0582-4cc3-945f-0229e086fad3-cni-binary-copy\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.981095 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.979867 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02929572-64d2-492c-8602-3a3dc9bc9701-serviceca\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.981095 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.980846 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/091076f1-c7ed-4f20-8191-2fbf472b2a92-ovn-node-metrics-cert\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.985833 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.985809 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:12:48.985833 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.985835 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:12:48.986001 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.985850 2555 projected.go:196] Error preparing data for projected volume kube-api-access-ph422 for pod openshift-network-diagnostics/network-check-target-68k9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:48.986001 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:48.985922 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422 podName:3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc nodeName:}" failed. No retries permitted until 2026-09-08 16:12:49.485902204 +0000 UTC m=+3.080900955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ph422" (UniqueName: "kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422") pod "network-check-target-68k9z" (UID: "3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:48.987949 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.987916 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-646d2\" (UniqueName: \"kubernetes.io/projected/091076f1-c7ed-4f20-8191-2fbf472b2a92-kube-api-access-646d2\") pod \"ovnkube-node-smsxf\" (UID: \"091076f1-c7ed-4f20-8191-2fbf472b2a92\") " pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:48.988120 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.988096 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8tgb\" (UniqueName: \"kubernetes.io/projected/266f7a5b-0582-4cc3-945f-0229e086fad3-kube-api-access-z8tgb\") pod \"multus-additional-cni-plugins-lj8pc\" (UID: \"266f7a5b-0582-4cc3-945f-0229e086fad3\") " pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:48.988871 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.988842 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8ll4\" (UniqueName: \"kubernetes.io/projected/02929572-64d2-492c-8602-3a3dc9bc9701-kube-api-access-z8ll4\") pod \"node-ca-nsbsw\" (UID: \"02929572-64d2-492c-8602-3a3dc9bc9701\") " pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:48.990095 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:48.990070 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzs7s\" (UniqueName: \"kubernetes.io/projected/51f04aaa-bb24-44ad-886b-f6ef20fd7dd2-kube-api-access-qzs7s\") pod \"iptables-alerter-lnzlp\" (UID: \"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2\") " pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:49.064682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.064651 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxdnw" Sep 08 16:12:49.071664 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.071456 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hz52h" Sep 08 16:12:49.074138 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:49.074107 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ff108a_6700_47c6_b566_f6e6740810e1.slice/crio-c05cb1b3e18ab65716e60194961056d78ba30657d83d14492a1433613ceb5b84 WatchSource:0}: Error finding container c05cb1b3e18ab65716e60194961056d78ba30657d83d14492a1433613ceb5b84: Status 404 returned error can't find the container with id c05cb1b3e18ab65716e60194961056d78ba30657d83d14492a1433613ceb5b84 Sep 08 16:12:49.078646 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.078389 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:12:49.083270 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.083014 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" Sep 08 16:12:49.083270 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:49.083100 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42de246_aa61_4934_96e1_07e4fb87f898.slice/crio-ba4e35be9aa6178c72830bdf10dcca54b3828ec7abc9ffc86f9a492fb063e716 WatchSource:0}: Error finding container ba4e35be9aa6178c72830bdf10dcca54b3828ec7abc9ffc86f9a492fb063e716: Status 404 returned error can't find the container with id ba4e35be9aa6178c72830bdf10dcca54b3828ec7abc9ffc86f9a492fb063e716 Sep 08 16:12:49.088814 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.088788 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" Sep 08 16:12:49.089204 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:49.089182 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64c5b01_186f_4002_827e_e2821aad92b0.slice/crio-5bc32de55fa61d7ded993c8a92d303a830f147782100f8bec96b957349dcf0e0 WatchSource:0}: Error finding container 5bc32de55fa61d7ded993c8a92d303a830f147782100f8bec96b957349dcf0e0: Status 404 returned error can't find the container with id 5bc32de55fa61d7ded993c8a92d303a830f147782100f8bec96b957349dcf0e0 Sep 08 16:12:49.094543 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:49.094516 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9142923_ec0a_459f_ab44_3594ebfa1c86.slice/crio-699b9568db99b3411bd01faba1139dc28d16b9e03c4f08117e3fdbe005412e72 WatchSource:0}: Error finding container 699b9568db99b3411bd01faba1139dc28d16b9e03c4f08117e3fdbe005412e72: Status 404 returned error can't find the container with id 699b9568db99b3411bd01faba1139dc28d16b9e03c4f08117e3fdbe005412e72 Sep 08 16:12:49.095451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.095431 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" Sep 08 16:12:49.098882 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:49.098857 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61544e0b_52f3_4842_8d62_ffcc0e2df2f1.slice/crio-a4c91771d271d83010838de871cff490887e5ca52c983a9578a2c258b8511542 WatchSource:0}: Error finding container a4c91771d271d83010838de871cff490887e5ca52c983a9578a2c258b8511542: Status 404 returned error can't find the container with id a4c91771d271d83010838de871cff490887e5ca52c983a9578a2c258b8511542 Sep 08 16:12:49.101824 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.101803 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:12:49.108087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.107583 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nsbsw" Sep 08 16:12:49.115863 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.113392 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnzlp" Sep 08 16:12:49.124132 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:49.124096 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02929572_64d2_492c_8602_3a3dc9bc9701.slice/crio-19de69e8b87223aed305e395d3993cc1ca78f6ce15cb000b6ec8d2e97e9af498 WatchSource:0}: Error finding container 19de69e8b87223aed305e395d3993cc1ca78f6ce15cb000b6ec8d2e97e9af498: Status 404 returned error can't find the container with id 19de69e8b87223aed305e395d3993cc1ca78f6ce15cb000b6ec8d2e97e9af498 Sep 08 16:12:49.136958 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:12:49.136920 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f04aaa_bb24_44ad_886b_f6ef20fd7dd2.slice/crio-9dff96cc1d6e803b7243ac28c9725bc920db82602d7d73a04161821a0e1828d7 WatchSource:0}: Error finding container 9dff96cc1d6e803b7243ac28c9725bc920db82602d7d73a04161821a0e1828d7: Status 404 returned error can't find the container with id 9dff96cc1d6e803b7243ac28c9725bc920db82602d7d73a04161821a0e1828d7 Sep 08 16:12:49.296717 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.296678 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:12:49.382817 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.382145 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:49.382817 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.382320 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:49.382817 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.382394 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:50.382372457 +0000 UTC m=+3.977371202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:49.482811 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.482773 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:49.483014 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.482954 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:49.483080 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.483043 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret podName:4234fc50-378f-4ac5-a183-0cb15a2365a5 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:50.483021808 +0000 UTC m=+4.078020568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret") pod "global-pull-secret-syncer-xtsbg" (UID: "4234fc50-378f-4ac5-a183-0cb15a2365a5") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:49.583341 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.583292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:49.583542 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.583511 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:12:49.583542 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.583533 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:12:49.583673 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.583548 2555 projected.go:196] Error preparing data for projected volume kube-api-access-ph422 for pod openshift-network-diagnostics/network-check-target-68k9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:49.583673 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.583641 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422 podName:3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc nodeName:}" failed. No retries permitted until 2026-09-08 16:12:50.58360381 +0000 UTC m=+4.178602552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph422" (UniqueName: "kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422") pod "network-check-target-68k9z" (UID: "3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:49.813039 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.812850 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:07:48 +0000 UTC" deadline="2028-05-12 11:48:30.595026325 +0000 UTC" Sep 08 16:12:49.813039 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.812928 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14683h35m40.782103397s" Sep 08 16:12:49.932029 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.931182 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:49.932029 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.931343 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:12:49.932029 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.931856 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:49.932029 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:49.931971 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:12:49.953032 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.952927 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerStarted","Data":"ae932383db5f1df2a934cfd523bc348466b87bf56673e5a06fa06a4e52fe153a"} Sep 08 16:12:49.955721 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.955574 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" event={"ID":"61544e0b-52f3-4842-8d62-ffcc0e2df2f1","Type":"ContainerStarted","Data":"a4c91771d271d83010838de871cff490887e5ca52c983a9578a2c258b8511542"} Sep 08 16:12:49.958864 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.958762 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" event={"ID":"a9142923-ec0a-459f-ab44-3594ebfa1c86","Type":"ContainerStarted","Data":"699b9568db99b3411bd01faba1139dc28d16b9e03c4f08117e3fdbe005412e72"} Sep 08 16:12:49.961765 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.961679 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tqbd2" event={"ID":"e64c5b01-186f-4002-827e-e2821aad92b0","Type":"ContainerStarted","Data":"5bc32de55fa61d7ded993c8a92d303a830f147782100f8bec96b957349dcf0e0"} Sep 08 16:12:49.964508 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.964430 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxdnw" event={"ID":"f0ff108a-6700-47c6-b566-f6e6740810e1","Type":"ContainerStarted","Data":"c05cb1b3e18ab65716e60194961056d78ba30657d83d14492a1433613ceb5b84"} Sep 08 16:12:49.967857 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.967768 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnzlp" event={"ID":"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2","Type":"ContainerStarted","Data":"9dff96cc1d6e803b7243ac28c9725bc920db82602d7d73a04161821a0e1828d7"} Sep 08 16:12:49.972444 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.972409 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nsbsw" event={"ID":"02929572-64d2-492c-8602-3a3dc9bc9701","Type":"ContainerStarted","Data":"19de69e8b87223aed305e395d3993cc1ca78f6ce15cb000b6ec8d2e97e9af498"} Sep 08 16:12:49.976103 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.976071 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"26546f3826ebdc21b5c729b5a5a477af82d2b98e55968a8dfeb12c5597a1b9aa"} Sep 08 16:12:49.980524 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:49.980491 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hz52h" event={"ID":"a42de246-aa61-4934-96e1-07e4fb87f898","Type":"ContainerStarted","Data":"ba4e35be9aa6178c72830bdf10dcca54b3828ec7abc9ffc86f9a492fb063e716"} Sep 08 16:12:50.362309 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:50.361969 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:12:50.391742 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:50.391699 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:50.392038 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.391898 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:50.392038 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.391964 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:52.391945887 +0000 UTC m=+5.986944625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:50.492732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:50.492688 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:50.492951 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.492931 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:50.493038 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.493016 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret podName:4234fc50-378f-4ac5-a183-0cb15a2365a5 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:52.492994588 +0000 UTC m=+6.087993342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret") pod "global-pull-secret-syncer-xtsbg" (UID: "4234fc50-378f-4ac5-a183-0cb15a2365a5") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:50.593658 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:50.593590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:50.593824 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.593799 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:12:50.593824 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.593821 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:12:50.593952 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.593833 2555 projected.go:196] Error preparing data for projected volume kube-api-access-ph422 for pod openshift-network-diagnostics/network-check-target-68k9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:50.593952 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.593891 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422 podName:3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc nodeName:}" failed. No retries permitted until 2026-09-08 16:12:52.593873954 +0000 UTC m=+6.188872718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph422" (UniqueName: "kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422") pod "network-check-target-68k9z" (UID: "3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:50.813962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:50.813865 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:07:48 +0000 UTC" deadline="2028-06-02 08:50:36.178081946 +0000 UTC" Sep 08 16:12:50.813962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:50.813912 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15184h37m45.364174252s" Sep 08 16:12:50.933480 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:50.933449 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:50.933671 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:50.933602 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:12:51.930687 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:51.930647 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:51.931184 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:51.930806 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:12:51.931529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:51.931507 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:51.931658 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:51.931633 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:12:52.413486 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:52.412719 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:52.413486 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.412872 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:52.413486 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.412941 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:56.41291955 +0000 UTC m=+10.007918292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:52.514126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:52.514088 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:52.514301 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.514266 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:52.514372 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.514336 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret podName:4234fc50-378f-4ac5-a183-0cb15a2365a5 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:56.514316394 +0000 UTC m=+10.109315132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret") pod "global-pull-secret-syncer-xtsbg" (UID: "4234fc50-378f-4ac5-a183-0cb15a2365a5") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:52.615141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:52.614538 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:52.615141 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.614719 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:12:52.615141 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.614742 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:12:52.615141 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.614755 2555 projected.go:196] Error preparing data for projected volume kube-api-access-ph422 for pod openshift-network-diagnostics/network-check-target-68k9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:52.615141 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.614813 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422 podName:3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc nodeName:}" failed. No retries permitted until 2026-09-08 16:12:56.614795832 +0000 UTC m=+10.209794574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph422" (UniqueName: "kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422") pod "network-check-target-68k9z" (UID: "3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:52.933168 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:52.933094 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:52.933694 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:52.933306 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:12:53.930423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:53.930391 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:53.930576 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:53.930522 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:12:53.930686 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:53.930568 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:53.930752 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:53.930722 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:12:54.931836 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:54.931375 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:54.931836 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:54.931502 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:12:55.931012 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:55.930925 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:55.931012 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:55.930977 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:55.931227 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:55.931053 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:12:55.931227 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:55.931188 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:12:56.445993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:56.445954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:56.446659 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.446076 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:56.446659 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.446156 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:04.446134037 +0000 UTC m=+18.041132780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:12:56.547395 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:56.546821 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:56.547395 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.546988 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:56.547395 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.547055 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret podName:4234fc50-378f-4ac5-a183-0cb15a2365a5 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:04.547036771 +0000 UTC m=+18.142035514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret") pod "global-pull-secret-syncer-xtsbg" (UID: "4234fc50-378f-4ac5-a183-0cb15a2365a5") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:12:56.647782 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:56.647704 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:56.647950 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.647907 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:12:56.647950 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.647937 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:12:56.647950 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.647951 2555 projected.go:196] Error preparing data for projected volume kube-api-access-ph422 for pod openshift-network-diagnostics/network-check-target-68k9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:56.648098 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.648015 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422 podName:3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc nodeName:}" failed. No retries permitted until 2026-09-08 16:13:04.647997284 +0000 UTC m=+18.242996038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph422" (UniqueName: "kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422") pod "network-check-target-68k9z" (UID: "3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:12:56.932338 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:56.931829 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:56.932338 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:56.931949 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:12:57.931309 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:57.930636 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:57.931309 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:57.930772 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:12:57.931309 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:57.931118 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:57.931309 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:57.931202 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:12:58.931049 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:58.931014 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:12:58.931369 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:58.931143 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:12:59.931093 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:59.931051 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:12:59.931294 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:59.931177 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:12:59.931294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:12:59.931194 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:12:59.931429 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:12:59.931320 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:00.930736 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:00.930691 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:00.930901 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:00.930847 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:01.930605 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:01.930559 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:01.931081 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:01.930694 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:01.931081 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:01.930761 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:01.931081 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:01.930885 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:02.930479 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:02.930446 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:02.930661 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:02.930586 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:03.931186 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:03.931146 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:03.931691 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:03.931144 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:03.931691 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:03.931317 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:03.931691 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:03.931387 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:04.506590 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:04.506546 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:04.506846 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.506675 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:13:04.506846 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.506750 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.506729634 +0000 UTC m=+34.101728373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:13:04.607258 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:04.607203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:04.607418 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.607345 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:13:04.607488 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.607420 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret podName:4234fc50-378f-4ac5-a183-0cb15a2365a5 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.607404297 +0000 UTC m=+34.202403035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret") pod "global-pull-secret-syncer-xtsbg" (UID: "4234fc50-378f-4ac5-a183-0cb15a2365a5") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:13:04.708445 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:04.708403 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:04.708633 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.708575 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:13:04.708633 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.708605 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:13:04.708633 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.708620 2555 projected.go:196] Error preparing data for projected volume kube-api-access-ph422 for pod openshift-network-diagnostics/network-check-target-68k9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:13:04.708760 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.708688 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422 podName:3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.708668444 +0000 UTC m=+34.303667198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph422" (UniqueName: "kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422") pod "network-check-target-68k9z" (UID: "3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:13:04.930989 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:04.930889 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:04.931161 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:04.931037 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:05.930652 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:05.930611 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:05.931096 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:05.930636 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:05.931096 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:05.930748 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:05.931096 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:05.930862 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:06.932187 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:06.932143 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:06.932657 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:06.932282 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:07.931051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:07.931019 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:07.931180 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:07.931151 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:07.931313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:07.931296 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:07.931432 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:07.931407 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:08.021932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.021898 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" event={"ID":"bcaf88a8105917bc3decd15ff63fd729","Type":"ContainerStarted","Data":"5f8e410bfdf1909cd54ca307146ab79ba42780cc03f50c98e6ea4decfe482b17"} Sep 08 16:13:08.025335 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.024633 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" event={"ID":"61544e0b-52f3-4842-8d62-ffcc0e2df2f1","Type":"ContainerStarted","Data":"c1005a6794df960b59ec63071f1d95e70007aa35bb38ba5df73a72bec617068f"} Sep 08 16:13:08.028997 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.028974 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"3fd3fe5573a24da0dff3f5f9385758da365e2e3542abb2609033a56e87ca6e09"} Sep 08 16:13:08.029105 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.029048 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"2f48dde18dc9704cdc86e10f46b6a04aef60cce3875a48959d6f1be6eca4a565"} Sep 08 16:13:08.029105 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.029078 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"f02e29c1b13a3d607f9581cb31d0c330faea8766ea0c85d5917244e94824221c"} Sep 08 16:13:08.029215 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.029109 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"cdffc99172e0bc6e2dfe1b28b2e6ebafde150eabccb85962504be062f873ca35"} Sep 08 16:13:08.030886 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.030860 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hz52h" event={"ID":"a42de246-aa61-4934-96e1-07e4fb87f898","Type":"ContainerStarted","Data":"70f5325cdb489ee7bcd94afbb0ccc1aba676ef517cf514559b4e3f48fc93ec21"} Sep 08 16:13:08.038539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.038471 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.ec2.internal" podStartSLOduration=20.038456641 podStartE2EDuration="20.038456641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:08.037692125 +0000 UTC m=+21.632690875" watchObservedRunningTime="2026-09-08 16:13:08.038456641 +0000 UTC m=+21.633455394" Sep 08 16:13:08.056779 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.056056 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xhlvb" podStartSLOduration=2.811430423 podStartE2EDuration="21.056034108s" totalImagesPullingTime="18.244603685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:08.055510413 +0000 UTC m=+21.650509198" watchObservedRunningTime="2026-09-08 16:13:08.056034108 +0000 UTC m=+21.651032868" Sep 08 16:13:08.072666 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.072615 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hz52h" podStartSLOduration=2.777392912 podStartE2EDuration="21.07259773s" totalImagesPullingTime="18.295204818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:08.072027498 +0000 UTC m=+21.667026316" watchObservedRunningTime="2026-09-08 16:13:08.07259773 +0000 UTC m=+21.667596488" Sep 08 16:13:08.931233 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:08.931193 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:08.931439 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:08.931356 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:09.034549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.034514 2555 generic.go:356] "Generic (PLEG): container finished" podID="266f7a5b-0582-4cc3-945f-0229e086fad3" containerID="a62c7e1080c686b8726af40297df8b65b476de3a034c7a6880e0b2a53193844e" exitCode=0 Sep 08 16:13:09.035169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.034588 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerDied","Data":"a62c7e1080c686b8726af40297df8b65b476de3a034c7a6880e0b2a53193844e"} Sep 08 16:13:09.036156 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.036135 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" event={"ID":"a9142923-ec0a-459f-ab44-3594ebfa1c86","Type":"ContainerStarted","Data":"039b31f528d35b655ca8f3283ad5eee6a013679ff4a3eb41f2a57add51bcb263"} Sep 08 16:13:09.037835 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.037811 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tqbd2" event={"ID":"e64c5b01-186f-4002-827e-e2821aad92b0","Type":"ContainerStarted","Data":"f332fb233cad2f7bde74b6a8898d0f61f4b0f2383d229e1750c137f002853943"} Sep 08 16:13:09.039424 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.039359 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxdnw" event={"ID":"f0ff108a-6700-47c6-b566-f6e6740810e1","Type":"ContainerStarted","Data":"e46001b2522479109d92cff41ee7c375428c5580f7ebede8cd7c9fc46c2e837f"} Sep 08 16:13:09.040955 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.040934 2555 generic.go:356] "Generic (PLEG): container finished" podID="7b512f4dee5198201752a4ab64dc951d" containerID="742f57d1f1c6c1bc554da7c7348661d69de78b2f1e0bf988bd411b1c955cad82" exitCode=0 Sep 08 16:13:09.041079 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.040996 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" event={"ID":"7b512f4dee5198201752a4ab64dc951d","Type":"ContainerDied","Data":"742f57d1f1c6c1bc554da7c7348661d69de78b2f1e0bf988bd411b1c955cad82"} Sep 08 16:13:09.043058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.043030 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnzlp" event={"ID":"51f04aaa-bb24-44ad-886b-f6ef20fd7dd2","Type":"ContainerStarted","Data":"8c3c9835925e2131303589f20d9105061a73518bb9ee8855ded40abdabc75d60"} Sep 08 16:13:09.044567 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.044539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nsbsw" event={"ID":"02929572-64d2-492c-8602-3a3dc9bc9701","Type":"ContainerStarted","Data":"a0c3fe67d8b1b5ed4c3d3f43775f893f03ead877b99c8366370b11e102115fe2"} Sep 08 16:13:09.047623 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.047602 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"d7f01b3dd0b05f9522224fd2b38147592ee57832d8322095986f290f0ac3cf80"} Sep 08 16:13:09.047720 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.047655 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"8b560c4c06d6099028a0c2667a098b6300ae48bd75d1b5dabeef2925424a0901"} Sep 08 16:13:09.066131 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.066081 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tqbd2" podStartSLOduration=3.866572336 podStartE2EDuration="22.066065836s" totalImagesPullingTime="18.1994935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:09.06577916 +0000 UTC m=+22.660777938" watchObservedRunningTime="2026-09-08 16:13:09.066065836 +0000 UTC m=+22.661064595" Sep 08 16:13:09.078064 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.078011 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lnzlp" podStartSLOduration=3.871806152 podStartE2EDuration="22.077994049s" totalImagesPullingTime="18.206187897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:09.077939649 +0000 UTC m=+22.672938428" watchObservedRunningTime="2026-09-08 16:13:09.077994049 +0000 UTC m=+22.672992808" Sep 08 16:13:09.091307 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.091237 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bxdnw" podStartSLOduration=4.1709424649999995 podStartE2EDuration="22.091216492s" totalImagesPullingTime="17.920274027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:09.090830885 +0000 UTC m=+22.685829668" watchObservedRunningTime="2026-09-08 16:13:09.091216492 +0000 UTC m=+22.686215253" Sep 08 16:13:09.108996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.108930 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nsbsw" podStartSLOduration=3.944828637 podStartE2EDuration="22.10890763s" totalImagesPullingTime="18.164078993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:09.10850073 +0000 UTC m=+22.703499490" watchObservedRunningTime="2026-09-08 16:13:09.10890763 +0000 UTC m=+22.703906391" Sep 08 16:13:09.222390 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.222359 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 16:13:09.824541 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.824429 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:13:09.222383284Z","UUID":"a5061b9d-71bd-49e5-866f-b74ec60316c4","Handler":null,"Name":"","Endpoint":""} Sep 08 16:13:09.825952 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.825929 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 16:13:09.826108 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.825966 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 16:13:09.930521 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.930491 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:09.930703 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:09.930528 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:09.930703 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:09.930655 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:09.930813 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:09.930792 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:10.052826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:10.052792 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" event={"ID":"a9142923-ec0a-459f-ab44-3594ebfa1c86","Type":"ContainerStarted","Data":"f29842adf9ac8ebded65c5fcc402fb6709a4effad7a6b3a87935ef2a2d455928"} Sep 08 16:13:10.055559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:10.055519 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" event={"ID":"7b512f4dee5198201752a4ab64dc951d","Type":"ContainerStarted","Data":"037b5d13c6ab1c770df0d3fae616cd772948e9718da0a352dc367614f950bd25"} Sep 08 16:13:10.070959 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:10.070893 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.ec2.internal" podStartSLOduration=22.070873292 podStartE2EDuration="22.070873292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:10.070414363 +0000 UTC m=+23.665413142" watchObservedRunningTime="2026-09-08 16:13:10.070873292 +0000 UTC m=+23.665872054" Sep 08 16:13:10.931230 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:10.931161 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:10.931379 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:10.931317 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:11.060081 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:11.060042 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" event={"ID":"a9142923-ec0a-459f-ab44-3594ebfa1c86","Type":"ContainerStarted","Data":"76fad406b3ba31ce31e4d2beb77d32c985eaa191cbaaefab475faa9ddc857440"} Sep 08 16:13:11.064007 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:11.063972 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"6d66c8cec898b798d540bba30afa16f4835b208943041c5d62e07a6ff2074387"} Sep 08 16:13:11.844138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:11.844092 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:13:11.844868 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:11.844846 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:13:11.858330 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:11.858273 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6c2h5" podStartSLOduration=3.989392245 podStartE2EDuration="24.858225009s" totalImagesPullingTime="20.868832764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:11.078631066 +0000 UTC m=+24.673629845" watchObservedRunningTime="2026-09-08 16:13:11.858225009 +0000 UTC m=+25.453223770" Sep 08 16:13:11.930511 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:11.930475 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:11.930710 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:11.930475 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:11.930710 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:11.930588 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:11.930833 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:11.930706 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:12.930617 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:12.930584 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:12.931081 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:12.930731 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:13.930724 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:13.930486 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:13.931302 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:13.930487 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:13.931302 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:13.930774 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:13.931302 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:13.930831 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:14.074372 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:14.074329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" event={"ID":"091076f1-c7ed-4f20-8191-2fbf472b2a92","Type":"ContainerStarted","Data":"4636b5e6929bbea0f8a9637e8ac9ee1e91a9adb7ac2b3703e84609e9156ea125"} Sep 08 16:13:14.074611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:14.074587 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:13:14.076289 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:14.076263 2555 generic.go:356] "Generic (PLEG): container finished" podID="266f7a5b-0582-4cc3-945f-0229e086fad3" containerID="f4d6e981893459c30630f47ca274dccfb3aef2c426f8c778f17fd855c5676993" exitCode=0 Sep 08 16:13:14.076397 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:14.076290 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerDied","Data":"f4d6e981893459c30630f47ca274dccfb3aef2c426f8c778f17fd855c5676993"} Sep 08 16:13:14.090941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:14.090914 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:13:14.104483 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:14.104427 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" podStartSLOduration=8.593592358 podStartE2EDuration="27.1044103s" totalImagesPullingTime="18.510817942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:14.102862183 +0000 UTC m=+27.697860932" watchObservedRunningTime="2026-09-08 16:13:14.1044103 +0000 UTC m=+27.699409059" Sep 08 16:13:14.930598 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:14.930562 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:14.930779 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:14.930732 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:15.079286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.079239 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:13:15.079514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.079335 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:13:15.094875 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.094849 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:13:15.278033 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.278003 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-68k9z"] Sep 08 16:13:15.278222 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.278128 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:15.278299 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:15.278221 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:15.281000 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.280971 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtsbg"] Sep 08 16:13:15.281153 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.281086 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:15.281227 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:15.281201 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:15.281647 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.281619 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fbd6c"] Sep 08 16:13:15.281769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:15.281730 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:15.281862 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:15.281835 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:16.083203 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.083159 2555 generic.go:356] "Generic (PLEG): container finished" podID="266f7a5b-0582-4cc3-945f-0229e086fad3" containerID="b50db1ae9886dd31b8df82b4b4487fe1d62033a8e989a41f02be0f559bd98c0c" exitCode=0 Sep 08 16:13:16.083596 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.083237 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerDied","Data":"b50db1ae9886dd31b8df82b4b4487fe1d62033a8e989a41f02be0f559bd98c0c"} Sep 08 16:13:16.609306 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.609274 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:13:16.609478 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.609444 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 16:13:16.610025 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.609995 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tqbd2" Sep 08 16:13:16.932620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.932537 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:16.932764 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.932636 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:16.932764 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:16.932686 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:16.932764 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:16.932702 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:16.932868 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:16.932766 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:16.932902 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:16.932873 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:18.092862 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:18.092771 2555 generic.go:356] "Generic (PLEG): container finished" podID="266f7a5b-0582-4cc3-945f-0229e086fad3" containerID="7db110630c2601aac54c8421f318e22c5491c2e5283a49ec791bf18571a35d70" exitCode=0 Sep 08 16:13:18.092862 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:18.092813 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerDied","Data":"7db110630c2601aac54c8421f318e22c5491c2e5283a49ec791bf18571a35d70"} Sep 08 16:13:18.931130 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:18.931037 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:18.931313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:18.931037 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:18.931313 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:18.931185 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:18.931313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:18.931231 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:18.931476 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:18.931322 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:18.931476 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:18.931376 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:20.530139 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:20.530100 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:20.530595 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.530275 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:13:20.530595 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.530350 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.530330849 +0000 UTC m=+66.125329593 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:13:20.631425 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:20.631380 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:20.631591 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.631513 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:13:20.631591 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.631577 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret podName:4234fc50-378f-4ac5-a183-0cb15a2365a5 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.631561535 +0000 UTC m=+66.226560271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret") pod "global-pull-secret-syncer-xtsbg" (UID: "4234fc50-378f-4ac5-a183-0cb15a2365a5") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:13:20.732547 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:20.732511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:20.732722 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.732642 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:13:20.732722 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.732657 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:13:20.732722 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.732668 2555 projected.go:196] Error preparing data for projected volume kube-api-access-ph422 for pod openshift-network-diagnostics/network-check-target-68k9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:13:20.732884 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.732727 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422 podName:3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.732713616 +0000 UTC m=+66.327712352 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph422" (UniqueName: "kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422") pod "network-check-target-68k9z" (UID: "3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:13:20.930451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:20.930421 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:20.930635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:20.930423 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:20.930635 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.930545 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:13:20.930635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:20.930423 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:20.930635 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.930628 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtsbg" podUID="4234fc50-378f-4ac5-a183-0cb15a2365a5" Sep 08 16:13:20.930815 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:20.930697 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68k9z" podUID="3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc" Sep 08 16:13:21.717610 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.717579 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-234.ec2.internal" event="NodeReady" Sep 08 16:13:21.718043 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.717725 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 16:13:21.762069 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.762032 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j4tfc"] Sep 08 16:13:21.786262 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.786209 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-765jc"] Sep 08 16:13:21.787171 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.786535 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:21.788816 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.788790 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 16:13:21.788970 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.788876 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 16:13:21.789237 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.789220 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlhzx\"" Sep 08 16:13:21.800093 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.800071 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j4tfc"] Sep 08 16:13:21.800217 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.800116 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-765jc"] Sep 08 16:13:21.800217 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.800199 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:21.802456 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.802432 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 16:13:21.802566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.802481 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7rjg\"" Sep 08 16:13:21.802566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.802482 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 16:13:21.802566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.802524 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 16:13:21.941126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.941082 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:21.941312 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.941148 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsz8\" (UniqueName: \"kubernetes.io/projected/b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008-kube-api-access-6rsz8\") pod \"ingress-canary-765jc\" (UID: \"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008\") " pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:21.941312 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.941185 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5881b84-4529-469e-ac2e-9ff4ca8726ef-tmp-dir\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:21.941312 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.941263 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9xh\" (UniqueName: \"kubernetes.io/projected/a5881b84-4529-469e-ac2e-9ff4ca8726ef-kube-api-access-lp9xh\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:21.941312 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.941302 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008-cert\") pod \"ingress-canary-765jc\" (UID: \"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008\") " pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:21.941517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:21.941331 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5881b84-4529-469e-ac2e-9ff4ca8726ef-config-volume\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.042426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.042343 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9xh\" (UniqueName: \"kubernetes.io/projected/a5881b84-4529-469e-ac2e-9ff4ca8726ef-kube-api-access-lp9xh\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.042426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.042394 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008-cert\") pod \"ingress-canary-765jc\" (UID: \"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008\") " pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:22.042631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.042429 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5881b84-4529-469e-ac2e-9ff4ca8726ef-config-volume\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.042631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.042458 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.042631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.042485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsz8\" (UniqueName: \"kubernetes.io/projected/b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008-kube-api-access-6rsz8\") pod \"ingress-canary-765jc\" (UID: \"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008\") " pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:22.042631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.042505 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5881b84-4529-469e-ac2e-9ff4ca8726ef-tmp-dir\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.042809 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:22.042778 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:13:22.042865 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:22.042857 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:13:22.542835984 +0000 UTC m=+36.137834723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:13:22.042928 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.042872 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5881b84-4529-469e-ac2e-9ff4ca8726ef-tmp-dir\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.043388 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.043357 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5881b84-4529-469e-ac2e-9ff4ca8726ef-config-volume\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.047327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.047299 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008-cert\") pod \"ingress-canary-765jc\" (UID: \"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008\") " pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:22.051148 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.051121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9xh\" (UniqueName: \"kubernetes.io/projected/a5881b84-4529-469e-ac2e-9ff4ca8726ef-kube-api-access-lp9xh\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.051423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.051395 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsz8\" (UniqueName: \"kubernetes.io/projected/b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008-kube-api-access-6rsz8\") pod \"ingress-canary-765jc\" (UID: \"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008\") " pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:22.109504 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.109470 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-765jc" Sep 08 16:13:22.544904 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.544872 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:22.545065 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:22.545022 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:13:22.545115 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:22.545088 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:13:23.54507382 +0000 UTC m=+37.140072561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:13:22.840180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.840102 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-765jc"] Sep 08 16:13:22.843381 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:13:22.843349 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e0fedb_4b6f_4a9d_b22c_22d4b7bbb008.slice/crio-418db4e2b5fecb1c1cd2507578b5c78ffae918cc3fc42b0bdf44c3b8d2f5aaf8 WatchSource:0}: Error finding container 418db4e2b5fecb1c1cd2507578b5c78ffae918cc3fc42b0bdf44c3b8d2f5aaf8: Status 404 returned error can't find the container with id 418db4e2b5fecb1c1cd2507578b5c78ffae918cc3fc42b0bdf44c3b8d2f5aaf8 Sep 08 16:13:22.933915 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.933891 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:22.934057 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.933979 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:22.934119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.934058 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:22.937309 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.937288 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 16:13:22.937452 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.937288 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:13:22.937452 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.937292 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rq8v\"" Sep 08 16:13:22.937452 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.937292 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74mht\"" Sep 08 16:13:22.937452 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.937336 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 16:13:22.937452 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:22.937336 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:13:23.110161 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:23.110128 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerStarted","Data":"abe4c061ef6ecf1ffa2eeb8877d7b5237522ce6afbe936a77a0a16b051fe1fb8"} Sep 08 16:13:23.111559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:23.111537 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-765jc" event={"ID":"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008","Type":"ContainerStarted","Data":"418db4e2b5fecb1c1cd2507578b5c78ffae918cc3fc42b0bdf44c3b8d2f5aaf8"} Sep 08 16:13:23.551263 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:23.551168 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:23.551397 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:23.551313 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:13:23.551397 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:23.551370 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:13:25.551355705 +0000 UTC m=+39.146354447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:13:24.117410 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:24.117186 2555 generic.go:356] "Generic (PLEG): container finished" podID="266f7a5b-0582-4cc3-945f-0229e086fad3" containerID="abe4c061ef6ecf1ffa2eeb8877d7b5237522ce6afbe936a77a0a16b051fe1fb8" exitCode=0 Sep 08 16:13:24.117990 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:24.117288 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerDied","Data":"abe4c061ef6ecf1ffa2eeb8877d7b5237522ce6afbe936a77a0a16b051fe1fb8"} Sep 08 16:13:25.123869 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:25.123784 2555 generic.go:356] "Generic (PLEG): container finished" podID="266f7a5b-0582-4cc3-945f-0229e086fad3" containerID="d11c6dde8ef0a183d04a38302a92c1218e52f2ab45f87c66b865ae2f6c804484" exitCode=0 Sep 08 16:13:25.123869 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:25.123854 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerDied","Data":"d11c6dde8ef0a183d04a38302a92c1218e52f2ab45f87c66b865ae2f6c804484"} Sep 08 16:13:25.568737 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:25.568703 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:25.568907 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:25.568869 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:13:25.568948 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:25.568936 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:13:29.568920979 +0000 UTC m=+43.163919716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:13:26.129396 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:26.129299 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" event={"ID":"266f7a5b-0582-4cc3-945f-0229e086fad3","Type":"ContainerStarted","Data":"d83eeb349e87a600083b941bb4d3285a3b2045e73b8667d1cbe74dc875cf766e"} Sep 08 16:13:26.130660 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:26.130637 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-765jc" event={"ID":"b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008","Type":"ContainerStarted","Data":"8a9b4eb340da4c9db9e9eaef77097554b0ff2801dfe785e4c6a4acb2b75ae0da"} Sep 08 16:13:26.152062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:26.152010 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lj8pc" podStartSLOduration=13.419423505 podStartE2EDuration="39.151993682s" totalImagesPullingTime="25.732570177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:26.151319949 +0000 UTC m=+39.746318728" watchObservedRunningTime="2026-09-08 16:13:26.151993682 +0000 UTC m=+39.746992440" Sep 08 16:13:26.172763 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:26.172716 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-765jc" podStartSLOduration=2.343684905 podStartE2EDuration="5.172698639s" totalImagesPullingTime="2.829013734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:26.172214146 +0000 UTC m=+39.767212898" watchObservedRunningTime="2026-09-08 16:13:26.172698639 +0000 UTC m=+39.767697398" Sep 08 16:13:29.597816 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:29.597764 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:29.598346 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:29.597936 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:13:29.598346 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:29.598036 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:13:37.598014829 +0000 UTC m=+51.193013565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:13:37.648656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:37.648597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:37.649111 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:37.648764 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:13:37.649111 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:37.648832 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:13:53.648815956 +0000 UTC m=+67.243814698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:13:47.058263 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.058217 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm"] Sep 08 16:13:47.091557 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.091520 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm"] Sep 08 16:13:47.091704 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.091626 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.095691 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.095663 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 16:13:47.095691 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.095691 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 16:13:47.095918 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.095719 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 16:13:47.095918 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.095674 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 08 16:13:47.095918 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.095797 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nzgws\"" Sep 08 16:13:47.105008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.104987 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smsxf" Sep 08 16:13:47.218406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.218364 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f36e387-c3ae-47c5-81cd-ace60aca2f38-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-694cc5f4dd-czffm\" (UID: \"4f36e387-c3ae-47c5-81cd-ace60aca2f38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.218622 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.218439 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ssm\" (UniqueName: \"kubernetes.io/projected/4f36e387-c3ae-47c5-81cd-ace60aca2f38-kube-api-access-w8ssm\") pod \"managed-serviceaccount-addon-agent-694cc5f4dd-czffm\" (UID: \"4f36e387-c3ae-47c5-81cd-ace60aca2f38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.319487 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.319395 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f36e387-c3ae-47c5-81cd-ace60aca2f38-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-694cc5f4dd-czffm\" (UID: \"4f36e387-c3ae-47c5-81cd-ace60aca2f38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.319487 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.319474 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ssm\" (UniqueName: \"kubernetes.io/projected/4f36e387-c3ae-47c5-81cd-ace60aca2f38-kube-api-access-w8ssm\") pod \"managed-serviceaccount-addon-agent-694cc5f4dd-czffm\" (UID: \"4f36e387-c3ae-47c5-81cd-ace60aca2f38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.322625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.322590 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f36e387-c3ae-47c5-81cd-ace60aca2f38-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-694cc5f4dd-czffm\" (UID: \"4f36e387-c3ae-47c5-81cd-ace60aca2f38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.327298 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.327274 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ssm\" (UniqueName: \"kubernetes.io/projected/4f36e387-c3ae-47c5-81cd-ace60aca2f38-kube-api-access-w8ssm\") pod \"managed-serviceaccount-addon-agent-694cc5f4dd-czffm\" (UID: \"4f36e387-c3ae-47c5-81cd-ace60aca2f38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.417149 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.417109 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:13:47.706644 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:47.706603 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm"] Sep 08 16:13:47.709948 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:13:47.709918 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f36e387_c3ae_47c5_81cd_ace60aca2f38.slice/crio-bdba99fd5f964d42127f954966e5bd387f60886634cf6e257f39b4b1f37beaa0 WatchSource:0}: Error finding container bdba99fd5f964d42127f954966e5bd387f60886634cf6e257f39b4b1f37beaa0: Status 404 returned error can't find the container with id bdba99fd5f964d42127f954966e5bd387f60886634cf6e257f39b4b1f37beaa0 Sep 08 16:13:48.185961 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:48.185924 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" event={"ID":"4f36e387-c3ae-47c5-81cd-ace60aca2f38","Type":"ContainerStarted","Data":"bdba99fd5f964d42127f954966e5bd387f60886634cf6e257f39b4b1f37beaa0"} Sep 08 16:13:51.194109 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:51.194072 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" event={"ID":"4f36e387-c3ae-47c5-81cd-ace60aca2f38","Type":"ContainerStarted","Data":"d7a4d516df2d42514faca5a5de041ef481d78f7b2a027729b9b471aec0f0a8b7"} Sep 08 16:13:51.209021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:51.208968 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" podStartSLOduration=0.898268648 podStartE2EDuration="4.208951438s" totalImagesPullingTime="3.31068279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:51.208163985 +0000 UTC m=+64.803162749" watchObservedRunningTime="2026-09-08 16:13:51.208951438 +0000 UTC m=+64.803950196" Sep 08 16:13:52.554732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.554687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:13:52.560081 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.560059 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:13:52.565607 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:52.565584 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 16:13:52.565664 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:52.565646 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:14:56.565628669 +0000 UTC m=+130.160627411 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : secret "metrics-daemon-secret" not found Sep 08 16:13:52.655906 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.655871 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:52.658417 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.658392 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:13:52.668357 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.668330 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4234fc50-378f-4ac5-a183-0cb15a2365a5-original-pull-secret\") pod \"global-pull-secret-syncer-xtsbg\" (UID: \"4234fc50-378f-4ac5-a183-0cb15a2365a5\") " pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:52.674333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.674314 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtsbg" Sep 08 16:13:52.756491 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.756451 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:52.759223 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.759202 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 16:13:52.769879 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.769844 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 16:13:52.780392 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.780366 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph422\" (UniqueName: \"kubernetes.io/projected/3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc-kube-api-access-ph422\") pod \"network-check-target-68k9z\" (UID: \"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc\") " pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:52.938111 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.938068 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtsbg"] Sep 08 16:13:52.941480 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:13:52.941456 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4234fc50_378f_4ac5_a183_0cb15a2365a5.slice/crio-a23223fb2eebc52f99cdb8f274606824fa0cb9be0bff4a4360cf7c4fa5b7869b WatchSource:0}: Error finding container a23223fb2eebc52f99cdb8f274606824fa0cb9be0bff4a4360cf7c4fa5b7869b: Status 404 returned error can't find the container with id a23223fb2eebc52f99cdb8f274606824fa0cb9be0bff4a4360cf7c4fa5b7869b Sep 08 16:13:52.955078 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.955053 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74mht\"" Sep 08 16:13:52.963253 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:52.963212 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:53.200305 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:53.200218 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtsbg" event={"ID":"4234fc50-378f-4ac5-a183-0cb15a2365a5","Type":"ContainerStarted","Data":"a23223fb2eebc52f99cdb8f274606824fa0cb9be0bff4a4360cf7c4fa5b7869b"} Sep 08 16:13:53.238511 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:53.238469 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-68k9z"] Sep 08 16:13:53.241816 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:13:53.241788 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3963b4ca_dc40_4bb1_af27_1c84b9dcbfdc.slice/crio-178b8b16f9929b62756f33323a558fd71b1f597d20885a7454b71d5b7ea5516e WatchSource:0}: Error finding container 178b8b16f9929b62756f33323a558fd71b1f597d20885a7454b71d5b7ea5516e: Status 404 returned error can't find the container with id 178b8b16f9929b62756f33323a558fd71b1f597d20885a7454b71d5b7ea5516e Sep 08 16:13:53.664255 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:53.664206 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:13:53.664697 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:53.664437 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:13:53.664697 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:13:53.664524 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:14:25.66450007 +0000 UTC m=+99.259498831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:13:54.204955 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:54.204885 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-68k9z" event={"ID":"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc","Type":"ContainerStarted","Data":"178b8b16f9929b62756f33323a558fd71b1f597d20885a7454b71d5b7ea5516e"} Sep 08 16:13:58.218880 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:58.218841 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-68k9z" event={"ID":"3963b4ca-dc40-4bb1-af27-1c84b9dcbfdc","Type":"ContainerStarted","Data":"e8f8c8461c3967f4cac26b542cec64eec82bb8f8dbd9c659c45a25eb9e6643fd"} Sep 08 16:13:58.219380 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:58.218943 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:13:58.220272 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:58.220233 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtsbg" event={"ID":"4234fc50-378f-4ac5-a183-0cb15a2365a5","Type":"ContainerStarted","Data":"1f4e76369f2475ee2b655721585234d3de10bbebf552eacb580dc45439bec4d3"} Sep 08 16:13:58.234109 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:58.234059 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-68k9z" podStartSLOduration=66.83976618 podStartE2EDuration="1m11.234044804s" totalImagesPullingTime="4.394278624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:58.233732773 +0000 UTC m=+71.828731545" watchObservedRunningTime="2026-09-08 16:13:58.234044804 +0000 UTC m=+71.829043557" Sep 08 16:13:58.247429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:13:58.247375 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xtsbg" podStartSLOduration=66.550382656 podStartE2EDuration="1m11.247360051s" totalImagesPullingTime="4.696977395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:58.246837185 +0000 UTC m=+71.841835944" watchObservedRunningTime="2026-09-08 16:13:58.247360051 +0000 UTC m=+71.842358810" Sep 08 16:14:25.684887 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:14:25.684840 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:14:25.685353 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:14:25.684953 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:14:25.685353 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:14:25.685015 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:15:29.685000687 +0000 UTC m=+163.279999425 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:14:29.225706 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:14:29.225668 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-68k9z" Sep 08 16:14:56.590049 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:14:56.589995 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:14:56.590568 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:14:56.590161 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 16:14:56.590568 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:14:56.590263 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs podName:3fb613c0-0100-414c-b958-e753b2d9a521 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:58.590227017 +0000 UTC m=+252.185225777 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs") pod "network-metrics-daemon-fbd6c" (UID: "3fb613c0-0100-414c-b958-e753b2d9a521") : secret "metrics-daemon-secret" not found Sep 08 16:15:24.797554 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:15:24.797506 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-j4tfc" podUID="a5881b84-4529-469e-ac2e-9ff4ca8726ef" Sep 08 16:15:25.436378 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:15:25.436345 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j4tfc" Sep 08 16:15:25.968830 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:15:25.968780 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fbd6c" podUID="3fb613c0-0100-414c-b958-e753b2d9a521" Sep 08 16:15:29.719324 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:15:29.719281 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:15:29.719733 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:15:29.719437 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 16:15:29.719733 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:15:29.719501 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls podName:a5881b84-4529-469e-ac2e-9ff4ca8726ef nodeName:}" failed. No retries permitted until 2026-09-08 16:17:31.719485366 +0000 UTC m=+285.314484103 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls") pod "dns-default-j4tfc" (UID: "a5881b84-4529-469e-ac2e-9ff4ca8726ef") : secret "dns-default-metrics-tls" not found Sep 08 16:15:36.932251 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:15:36.932204 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:15:51.498500 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:15:51.498413 2555 generic.go:356] "Generic (PLEG): container finished" podID="4f36e387-c3ae-47c5-81cd-ace60aca2f38" containerID="d7a4d516df2d42514faca5a5de041ef481d78f7b2a027729b9b471aec0f0a8b7" exitCode=255 Sep 08 16:15:51.498500 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:15:51.498467 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" event={"ID":"4f36e387-c3ae-47c5-81cd-ace60aca2f38","Type":"ContainerDied","Data":"d7a4d516df2d42514faca5a5de041ef481d78f7b2a027729b9b471aec0f0a8b7"} Sep 08 16:15:51.504172 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:15:51.504132 2555 scope.go:117] "RemoveContainer" containerID="d7a4d516df2d42514faca5a5de041ef481d78f7b2a027729b9b471aec0f0a8b7" Sep 08 16:15:52.503093 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:15:52.503059 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" event={"ID":"4f36e387-c3ae-47c5-81cd-ace60aca2f38","Type":"ContainerStarted","Data":"80fd9b3925d85a57060f0acebb9d15be7332650c1c736d3f577b7a9424c0d9ec"} Sep 08 16:16:34.617394 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.617358 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7"] Sep 08 16:16:34.619234 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.619216 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" Sep 08 16:16:34.622292 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.622266 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 08 16:16:34.622413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.622382 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:16:34.624330 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.624309 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b7z2f\"" Sep 08 16:16:34.628069 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.628016 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs"] Sep 08 16:16:34.629895 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.629870 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-srwhq"] Sep 08 16:16:34.630074 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.630055 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.631819 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.631790 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48"] Sep 08 16:16:34.631919 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.631906 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.633646 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.633627 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jfn66\"" Sep 08 16:16:34.633744 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.633628 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 08 16:16:34.633744 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.633698 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j"] Sep 08 16:16:34.633864 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.633849 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.634446 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.634379 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 08 16:16:34.634551 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.634536 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 16:16:34.634625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.634558 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 16:16:34.634886 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.634862 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 08 16:16:34.634993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.634914 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 08 16:16:34.634993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.634953 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:16:34.635113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.635052 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 08 16:16:34.635113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.635057 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8vbq2\"" Sep 08 16:16:34.635946 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.635929 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7"] Sep 08 16:16:34.636035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.636010 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.636423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.636386 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:16:34.636837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.636819 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 08 16:16:34.637398 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.637376 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 08 16:16:34.637890 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.637874 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 08 16:16:34.637966 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.637878 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jhn82\"" Sep 08 16:16:34.642662 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.642639 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 08 16:16:34.643172 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.643149 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wb4mg\"" Sep 08 16:16:34.643282 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.643265 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 16:16:34.643506 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.643482 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 16:16:34.644474 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.644453 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 08 16:16:34.645717 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.645682 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs"] Sep 08 16:16:34.647556 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.647531 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 08 16:16:34.647917 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.647896 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-srwhq"] Sep 08 16:16:34.650840 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.650814 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48"] Sep 08 16:16:34.656976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.656956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5n7t\" (UniqueName: \"kubernetes.io/projected/5166965d-9ee0-4e8a-b74f-ac3c9735bfcc-kube-api-access-h5n7t\") pod \"volume-data-source-validator-695c895bd6-qzbg7\" (UID: \"5166965d-9ee0-4e8a-b74f-ac3c9735bfcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" Sep 08 16:16:34.659168 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.659146 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j"] Sep 08 16:16:34.712082 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.712045 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f9dn5"] Sep 08 16:16:34.714039 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.714019 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:34.716502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.716472 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 08 16:16:34.716613 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.716554 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cmc99\"" Sep 08 16:16:34.716798 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.716782 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 08 16:16:34.724106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.724083 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb"] Sep 08 16:16:34.725898 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.725879 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-844f884987-42b4w"] Sep 08 16:16:34.726009 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.725991 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:34.727730 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.727711 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.728367 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.728350 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-85mxb\"" Sep 08 16:16:34.728771 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.728752 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f9dn5"] Sep 08 16:16:34.735306 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.735284 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 16:16:34.735415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.735302 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgmfx\"" Sep 08 16:16:34.735583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.735565 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:16:34.735914 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.735897 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 08 16:16:34.736102 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.736086 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 16:16:34.736171 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.736107 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 16:16:34.742918 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.742896 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 08 16:16:34.744008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.743989 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 16:16:34.749327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.749304 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb"] Sep 08 16:16:34.755372 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.755344 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-844f884987-42b4w"] Sep 08 16:16:34.757518 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757495 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee82cb26-8982-49a1-9d19-03981de45ab9-serving-cert\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.757648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757532 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bncsb\" (UniqueName: \"kubernetes.io/projected/ee82cb26-8982-49a1-9d19-03981de45ab9-kube-api-access-bncsb\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.757648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757562 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-nginx-conf\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:34.757648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757597 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e9c00eb-2239-4e76-9155-8bb822586051-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.757806 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757692 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a5a8ea-d576-42a5-a612-fd894d234b40-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.757806 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757729 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:34.757806 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757778 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b979f2af-e275-4d96-a9aa-4c5d72a1410e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.757920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757804 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a5a8ea-d576-42a5-a612-fd894d234b40-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.757920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757828 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e9c00eb-2239-4e76-9155-8bb822586051-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.757920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757873 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g92c\" (UniqueName: \"kubernetes.io/projected/a1a5a8ea-d576-42a5-a612-fd894d234b40-kube-api-access-7g92c\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.757920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757905 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4pq\" (UniqueName: \"kubernetes.io/projected/1e9c00eb-2239-4e76-9155-8bb822586051-kube-api-access-cj4pq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.758058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757936 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:34.758058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757951 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee82cb26-8982-49a1-9d19-03981de45ab9-config\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.758058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.757994 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5n7t\" (UniqueName: \"kubernetes.io/projected/5166965d-9ee0-4e8a-b74f-ac3c9735bfcc-kube-api-access-h5n7t\") pod \"volume-data-source-validator-695c895bd6-qzbg7\" (UID: \"5166965d-9ee0-4e8a-b74f-ac3c9735bfcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" Sep 08 16:16:34.758058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.758029 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.758058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.758047 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a8ea-d576-42a5-a612-fd894d234b40-snapshots\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.758443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.758094 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9655\" (UniqueName: \"kubernetes.io/projected/b979f2af-e275-4d96-a9aa-4c5d72a1410e-kube-api-access-h9655\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.758443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.758116 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a8ea-d576-42a5-a612-fd894d234b40-tmp\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.758443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.758137 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsldn\" (UniqueName: \"kubernetes.io/projected/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-kube-api-access-bsldn\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:34.758443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.758161 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1a5a8ea-d576-42a5-a612-fd894d234b40-serving-cert\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.771338 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.771309 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5n7t\" (UniqueName: \"kubernetes.io/projected/5166965d-9ee0-4e8a-b74f-ac3c9735bfcc-kube-api-access-h5n7t\") pod \"volume-data-source-validator-695c895bd6-qzbg7\" (UID: \"5166965d-9ee0-4e8a-b74f-ac3c9735bfcc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" Sep 08 16:16:34.859460 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859404 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:34.859460 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859452 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee82cb26-8982-49a1-9d19-03981de45ab9-config\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859475 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a8ea-d576-42a5-a612-fd894d234b40-snapshots\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859507 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859538 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgbl\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-kube-api-access-wcgbl\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859568 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9655\" (UniqueName: \"kubernetes.io/projected/b979f2af-e275-4d96-a9aa-4c5d72a1410e-kube-api-access-h9655\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.859577 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.859626 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a8ea-d576-42a5-a612-fd894d234b40-tmp\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.859712 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.859663 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls podName:492953d2-fe38-47ba-aec4-d6a8a70c4dc5 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:35.359643391 +0000 UTC m=+228.954642149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6fbgb" (UID: "492953d2-fe38-47ba-aec4-d6a8a70c4dc5") : secret "samples-operator-tls" not found Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1a5a8ea-d576-42a5-a612-fd894d234b40-serving-cert\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859815 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee82cb26-8982-49a1-9d19-03981de45ab9-serving-cert\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859839 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-installation-pull-secrets\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859862 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e9c00eb-2239-4e76-9155-8bb822586051-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.859889 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls podName:b979f2af-e275-4d96-a9aa-4c5d72a1410e nodeName:}" failed. No retries permitted until 2026-09-08 16:16:35.359872439 +0000 UTC m=+228.954871176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vk2j" (UID: "b979f2af-e275-4d96-a9aa-4c5d72a1410e") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859921 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a5a8ea-d576-42a5-a612-fd894d234b40-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859948 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.859986 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a5a8ea-d576-42a5-a612-fd894d234b40-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860012 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-certificates\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860036 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-trusted-ca\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860065 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g92c\" (UniqueName: \"kubernetes.io/projected/a1a5a8ea-d576-42a5-a612-fd894d234b40-kube-api-access-7g92c\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.860099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860092 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4pq\" (UniqueName: \"kubernetes.io/projected/1e9c00eb-2239-4e76-9155-8bb822586051-kube-api-access-cj4pq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860115 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a8ea-d576-42a5-a612-fd894d234b40-tmp\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860129 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-ca-trust-extracted\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860150 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee82cb26-8982-49a1-9d19-03981de45ab9-config\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860214 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a8ea-d576-42a5-a612-fd894d234b40-snapshots\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.860409 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsldn\" (UniqueName: \"kubernetes.io/projected/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-kube-api-access-bsldn\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.860470 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert podName:aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa nodeName:}" failed. No retries permitted until 2026-09-08 16:16:35.360453303 +0000 UTC m=+228.955452057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f9dn5" (UID: "aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa") : secret "networking-console-plugin-cert" not found Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860477 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e9c00eb-2239-4e76-9155-8bb822586051-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860492 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860630 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bncsb\" (UniqueName: \"kubernetes.io/projected/ee82cb26-8982-49a1-9d19-03981de45ab9-kube-api-access-bncsb\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860644 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a5a8ea-d576-42a5-a612-fd894d234b40-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860685 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-nginx-conf\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860707 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b979f2af-e275-4d96-a9aa-4c5d72a1410e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.860719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860726 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e9c00eb-2239-4e76-9155-8bb822586051-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.861460 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860786 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-image-registry-private-configuration\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.861460 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.860813 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-bound-sa-token\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.861460 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.861402 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a5a8ea-d576-42a5-a612-fd894d234b40-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.861572 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.861466 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-nginx-conf\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:34.861572 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.861477 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b979f2af-e275-4d96-a9aa-4c5d72a1410e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.862578 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.862549 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee82cb26-8982-49a1-9d19-03981de45ab9-serving-cert\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.862678 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.862654 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1a5a8ea-d576-42a5-a612-fd894d234b40-serving-cert\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.862948 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.862932 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e9c00eb-2239-4e76-9155-8bb822586051-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.875810 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.875740 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bncsb\" (UniqueName: \"kubernetes.io/projected/ee82cb26-8982-49a1-9d19-03981de45ab9-kube-api-access-bncsb\") pod \"service-ca-operator-859f94c4b-clpvs\" (UID: \"ee82cb26-8982-49a1-9d19-03981de45ab9\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.875921 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.875842 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4pq\" (UniqueName: \"kubernetes.io/projected/1e9c00eb-2239-4e76-9155-8bb822586051-kube-api-access-cj4pq\") pod \"kube-storage-version-migrator-operator-8d7dd4675-65p48\" (UID: \"1e9c00eb-2239-4e76-9155-8bb822586051\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.877977 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.877949 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsldn\" (UniqueName: \"kubernetes.io/projected/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-kube-api-access-bsldn\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:34.881539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.881517 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g92c\" (UniqueName: \"kubernetes.io/projected/a1a5a8ea-d576-42a5-a612-fd894d234b40-kube-api-access-7g92c\") pod \"insights-operator-76bcd6ccf5-srwhq\" (UID: \"a1a5a8ea-d576-42a5-a612-fd894d234b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.889512 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.889485 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9655\" (UniqueName: \"kubernetes.io/projected/b979f2af-e275-4d96-a9aa-4c5d72a1410e-kube-api-access-h9655\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:34.927943 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.927903 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" Sep 08 16:16:34.942919 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.942890 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" Sep 08 16:16:34.952384 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.952362 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" Sep 08 16:16:34.959480 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.959357 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" Sep 08 16:16:34.961327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961212 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgbl\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-kube-api-access-wcgbl\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961296 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-installation-pull-secrets\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961356 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-certificates\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961382 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-trusted-ca\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961418 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-ca-trust-extracted\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961727 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961469 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961727 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961504 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-image-registry-private-configuration\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961727 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.961530 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-bound-sa-token\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.961967 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.961895 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:16:34.961967 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.961920 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-844f884987-42b4w: secret "image-registry-tls" not found Sep 08 16:16:34.962079 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:34.961994 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls podName:b3c44751-cd04-4aa8-9bdf-3cf9dad048d2 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:35.461973143 +0000 UTC m=+229.056971894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls") pod "image-registry-844f884987-42b4w" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2") : secret "image-registry-tls" not found Sep 08 16:16:34.962140 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.962089 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-ca-trust-extracted\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.962563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.962538 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-certificates\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.962857 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.962810 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-trusted-ca\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.964945 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.964917 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-installation-pull-secrets\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.965332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.965306 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-image-registry-private-configuration\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.975041 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.974980 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-bound-sa-token\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:34.976546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:34.976494 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgbl\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-kube-api-access-wcgbl\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:35.214491 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.214465 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7"] Sep 08 16:16:35.217290 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:16:35.217255 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5166965d_9ee0_4e8a_b74f_ac3c9735bfcc.slice/crio-ca50f6f121560ebd4efda035e7801bf4e72b7182f85448192c46b398e086d3e5 WatchSource:0}: Error finding container ca50f6f121560ebd4efda035e7801bf4e72b7182f85448192c46b398e086d3e5: Status 404 returned error can't find the container with id ca50f6f121560ebd4efda035e7801bf4e72b7182f85448192c46b398e086d3e5 Sep 08 16:16:35.231882 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.231853 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs"] Sep 08 16:16:35.234565 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:16:35.234539 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee82cb26_8982_49a1_9d19_03981de45ab9.slice/crio-5a5c0c6557f7e393b3fd29b918f4f5319704e082c3c1c2be8b1ba8a4bd4a1eb3 WatchSource:0}: Error finding container 5a5c0c6557f7e393b3fd29b918f4f5319704e082c3c1c2be8b1ba8a4bd4a1eb3: Status 404 returned error can't find the container with id 5a5c0c6557f7e393b3fd29b918f4f5319704e082c3c1c2be8b1ba8a4bd4a1eb3 Sep 08 16:16:35.239695 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.239669 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-srwhq"] Sep 08 16:16:35.241373 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:16:35.241340 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a5a8ea_d576_42a5_a612_fd894d234b40.slice/crio-8a4042fcb0a88da836be725e5716349349db28d246f565da3e3e7c30282223db WatchSource:0}: Error finding container 8a4042fcb0a88da836be725e5716349349db28d246f565da3e3e7c30282223db: Status 404 returned error can't find the container with id 8a4042fcb0a88da836be725e5716349349db28d246f565da3e3e7c30282223db Sep 08 16:16:35.260022 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.259989 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48"] Sep 08 16:16:35.262917 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:16:35.262895 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9c00eb_2239_4e76_9155_8bb822586051.slice/crio-7f7f795112baaee1b6d9b0f530d2822c50cca93ef726c3a6bd29ad58ef71d237 WatchSource:0}: Error finding container 7f7f795112baaee1b6d9b0f530d2822c50cca93ef726c3a6bd29ad58ef71d237: Status 404 returned error can't find the container with id 7f7f795112baaee1b6d9b0f530d2822c50cca93ef726c3a6bd29ad58ef71d237 Sep 08 16:16:35.365533 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.365472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:35.365533 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.365537 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:35.365790 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.365586 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:35.365790 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.365637 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:16:35.365790 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.365687 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:35.365790 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.365701 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 16:16:35.365790 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.365703 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls podName:492953d2-fe38-47ba-aec4-d6a8a70c4dc5 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:36.365687321 +0000 UTC m=+229.960686058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6fbgb" (UID: "492953d2-fe38-47ba-aec4-d6a8a70c4dc5") : secret "samples-operator-tls" not found Sep 08 16:16:35.365790 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.365750 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls podName:b979f2af-e275-4d96-a9aa-4c5d72a1410e nodeName:}" failed. No retries permitted until 2026-09-08 16:16:36.365738088 +0000 UTC m=+229.960736825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vk2j" (UID: "b979f2af-e275-4d96-a9aa-4c5d72a1410e") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:35.365790 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.365766 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert podName:aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa nodeName:}" failed. No retries permitted until 2026-09-08 16:16:36.365760087 +0000 UTC m=+229.960758823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f9dn5" (UID: "aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa") : secret "networking-console-plugin-cert" not found Sep 08 16:16:35.466671 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.466590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:35.466776 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.466740 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:16:35.466776 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.466759 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-844f884987-42b4w: secret "image-registry-tls" not found Sep 08 16:16:35.466852 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:35.466813 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls podName:b3c44751-cd04-4aa8-9bdf-3cf9dad048d2 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:36.466797942 +0000 UTC m=+230.061796678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls") pod "image-registry-844f884987-42b4w" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2") : secret "image-registry-tls" not found Sep 08 16:16:35.609100 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.609066 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" event={"ID":"a1a5a8ea-d576-42a5-a612-fd894d234b40","Type":"ContainerStarted","Data":"8a4042fcb0a88da836be725e5716349349db28d246f565da3e3e7c30282223db"} Sep 08 16:16:35.610344 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.610315 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" event={"ID":"1e9c00eb-2239-4e76-9155-8bb822586051","Type":"ContainerStarted","Data":"7f7f795112baaee1b6d9b0f530d2822c50cca93ef726c3a6bd29ad58ef71d237"} Sep 08 16:16:35.611434 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.611409 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" event={"ID":"ee82cb26-8982-49a1-9d19-03981de45ab9","Type":"ContainerStarted","Data":"5a5c0c6557f7e393b3fd29b918f4f5319704e082c3c1c2be8b1ba8a4bd4a1eb3"} Sep 08 16:16:35.612402 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:35.612383 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" event={"ID":"5166965d-9ee0-4e8a-b74f-ac3c9735bfcc","Type":"ContainerStarted","Data":"ca50f6f121560ebd4efda035e7801bf4e72b7182f85448192c46b398e086d3e5"} Sep 08 16:16:36.373330 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:36.373288 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:36.373822 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:36.373351 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:36.373822 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:36.373408 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:36.373822 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.373481 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:16:36.373822 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.373562 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls podName:492953d2-fe38-47ba-aec4-d6a8a70c4dc5 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:38.373537501 +0000 UTC m=+231.968536251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6fbgb" (UID: "492953d2-fe38-47ba-aec4-d6a8a70c4dc5") : secret "samples-operator-tls" not found Sep 08 16:16:36.373822 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.373596 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 16:16:36.373822 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.373652 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert podName:aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa nodeName:}" failed. No retries permitted until 2026-09-08 16:16:38.373635938 +0000 UTC m=+231.968634678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f9dn5" (UID: "aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa") : secret "networking-console-plugin-cert" not found Sep 08 16:16:36.373822 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.373713 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:36.374193 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.373746 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls podName:b979f2af-e275-4d96-a9aa-4c5d72a1410e nodeName:}" failed. No retries permitted until 2026-09-08 16:16:38.373735675 +0000 UTC m=+231.968734415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vk2j" (UID: "b979f2af-e275-4d96-a9aa-4c5d72a1410e") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:36.474622 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:36.474582 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:36.474807 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.474756 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:16:36.474807 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.474773 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-844f884987-42b4w: secret "image-registry-tls" not found Sep 08 16:16:36.474911 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:36.474832 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls podName:b3c44751-cd04-4aa8-9bdf-3cf9dad048d2 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:38.47481353 +0000 UTC m=+232.069812273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls") pod "image-registry-844f884987-42b4w" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2") : secret "image-registry-tls" not found Sep 08 16:16:38.390027 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.389991 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.390051 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.390101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.390172 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.390228 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.390269 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls podName:492953d2-fe38-47ba-aec4-d6a8a70c4dc5 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:42.390232983 +0000 UTC m=+235.985231725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6fbgb" (UID: "492953d2-fe38-47ba-aec4-d6a8a70c4dc5") : secret "samples-operator-tls" not found Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.390293 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.390313 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls podName:b979f2af-e275-4d96-a9aa-4c5d72a1410e nodeName:}" failed. No retries permitted until 2026-09-08 16:16:42.390296476 +0000 UTC m=+235.985295232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vk2j" (UID: "b979f2af-e275-4d96-a9aa-4c5d72a1410e") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:38.390684 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.390338 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert podName:aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa nodeName:}" failed. No retries permitted until 2026-09-08 16:16:42.390324918 +0000 UTC m=+235.985323669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f9dn5" (UID: "aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa") : secret "networking-console-plugin-cert" not found Sep 08 16:16:38.490836 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.490799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:38.491006 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.490975 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:16:38.491006 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.490998 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-844f884987-42b4w: secret "image-registry-tls" not found Sep 08 16:16:38.491084 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:38.491056 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls podName:b3c44751-cd04-4aa8-9bdf-3cf9dad048d2 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:42.491040895 +0000 UTC m=+236.086039632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls") pod "image-registry-844f884987-42b4w" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2") : secret "image-registry-tls" not found Sep 08 16:16:38.627951 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.627837 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" event={"ID":"a1a5a8ea-d576-42a5-a612-fd894d234b40","Type":"ContainerStarted","Data":"db80ae435ced91d45c81702077ffa60d9aec767d0a884f4474add711d50af52e"} Sep 08 16:16:38.632366 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.632275 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" event={"ID":"1e9c00eb-2239-4e76-9155-8bb822586051","Type":"ContainerStarted","Data":"e9b75dffcc7a789ae94c6568b1a1ad39b73a1f27256650d04794d8d68f7676e0"} Sep 08 16:16:38.635223 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.635038 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" event={"ID":"ee82cb26-8982-49a1-9d19-03981de45ab9","Type":"ContainerStarted","Data":"06f4c5c120d0bff8a62b84a6901967c722038f6a6edddea7d5821b7bcffbcaa4"} Sep 08 16:16:38.637428 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.637205 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" event={"ID":"5166965d-9ee0-4e8a-b74f-ac3c9735bfcc","Type":"ContainerStarted","Data":"1367020260e65696cdc60029c15a143f20df222651a5be160f53c468a165504d"} Sep 08 16:16:38.651303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.651153 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" podStartSLOduration=1.365061996 podStartE2EDuration="4.651131365s" totalImagesPullingTime="3.286069369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:38.649361072 +0000 UTC m=+232.244359846" watchObservedRunningTime="2026-09-08 16:16:38.651131365 +0000 UTC m=+232.246130125" Sep 08 16:16:38.680390 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.680314 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-qzbg7" podStartSLOduration=1.372358877 podStartE2EDuration="4.680291944s" totalImagesPullingTime="3.307933067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:38.68001946 +0000 UTC m=+232.275018220" watchObservedRunningTime="2026-09-08 16:16:38.680291944 +0000 UTC m=+232.275290704" Sep 08 16:16:38.709913 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.709855 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" podStartSLOduration=1.44090473 podStartE2EDuration="4.709837685s" totalImagesPullingTime="3.268932955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:38.709217136 +0000 UTC m=+232.304215895" watchObservedRunningTime="2026-09-08 16:16:38.709837685 +0000 UTC m=+232.304836440" Sep 08 16:16:38.735112 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:38.735056 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" podStartSLOduration=1.4374198630000001 podStartE2EDuration="4.735037458s" totalImagesPullingTime="3.297617595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:38.734466022 +0000 UTC m=+232.329464806" watchObservedRunningTime="2026-09-08 16:16:38.735037458 +0000 UTC m=+232.330036217" Sep 08 16:16:39.597153 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.597101 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92"] Sep 08 16:16:39.600566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.600493 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" Sep 08 16:16:39.602932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.602908 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2gnpg\"" Sep 08 16:16:39.603223 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.603203 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 08 16:16:39.603860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.603839 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 08 16:16:39.613491 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.613454 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92"] Sep 08 16:16:39.694495 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.694462 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d5px9"] Sep 08 16:16:39.697724 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.697693 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.700303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.700273 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 16:16:39.700303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.700291 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mlgpx\"" Sep 08 16:16:39.700713 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.700696 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 16:16:39.700884 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.700863 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhvg\" (UniqueName: \"kubernetes.io/projected/0cd2d9b3-c508-4471-b728-d5faf5bcf5b1-kube-api-access-pwhvg\") pod \"migrator-fdbb8fdff-5mj92\" (UID: \"0cd2d9b3-c508-4471-b728-d5faf5bcf5b1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" Sep 08 16:16:39.711188 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.711164 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d5px9"] Sep 08 16:16:39.801481 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.801441 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.801481 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.801480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd4d2\" (UniqueName: \"kubernetes.io/projected/f4bed502-a9da-4bcb-a714-256171637ea8-kube-api-access-rd4d2\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.801742 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.801511 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhvg\" (UniqueName: \"kubernetes.io/projected/0cd2d9b3-c508-4471-b728-d5faf5bcf5b1-kube-api-access-pwhvg\") pod \"migrator-fdbb8fdff-5mj92\" (UID: \"0cd2d9b3-c508-4471-b728-d5faf5bcf5b1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" Sep 08 16:16:39.801742 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.801547 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.801742 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.801643 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4bed502-a9da-4bcb-a714-256171637ea8-crio-socket\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.801742 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.801680 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4bed502-a9da-4bcb-a714-256171637ea8-data-volume\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.810479 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.810451 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhvg\" (UniqueName: \"kubernetes.io/projected/0cd2d9b3-c508-4471-b728-d5faf5bcf5b1-kube-api-access-pwhvg\") pod \"migrator-fdbb8fdff-5mj92\" (UID: \"0cd2d9b3-c508-4471-b728-d5faf5bcf5b1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" Sep 08 16:16:39.902594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.902480 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.902594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.902536 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4d2\" (UniqueName: \"kubernetes.io/projected/f4bed502-a9da-4bcb-a714-256171637ea8-kube-api-access-rd4d2\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.902894 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:39.902627 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:16:39.902894 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.902691 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.902894 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:39.902702 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:40.402686824 +0000 UTC m=+233.997685561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : configmap "kube-rbac-proxy" not found Sep 08 16:16:39.902894 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.902778 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4bed502-a9da-4bcb-a714-256171637ea8-crio-socket\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.902894 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.902816 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4bed502-a9da-4bcb-a714-256171637ea8-data-volume\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.902894 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:39.902846 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:16:39.902894 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.902850 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f4bed502-a9da-4bcb-a714-256171637ea8-crio-socket\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.903151 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:39.902915 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:40.402896946 +0000 UTC m=+233.997895702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : secret "insights-runtime-extractor-tls" not found Sep 08 16:16:39.903191 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.903179 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4bed502-a9da-4bcb-a714-256171637ea8-data-volume\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:39.911052 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.911030 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" Sep 08 16:16:39.911418 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:39.911396 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd4d2\" (UniqueName: \"kubernetes.io/projected/f4bed502-a9da-4bcb-a714-256171637ea8-kube-api-access-rd4d2\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:40.192060 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:40.192031 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92"] Sep 08 16:16:40.194731 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:16:40.194699 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd2d9b3_c508_4471_b728_d5faf5bcf5b1.slice/crio-b6819c6aeaa633819f77c98d388747aa0a8fce0e8ba36c64ee8e94b27307c5fc WatchSource:0}: Error finding container b6819c6aeaa633819f77c98d388747aa0a8fce0e8ba36c64ee8e94b27307c5fc: Status 404 returned error can't find the container with id b6819c6aeaa633819f77c98d388747aa0a8fce0e8ba36c64ee8e94b27307c5fc Sep 08 16:16:40.406545 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:40.406500 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:40.406735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:40.406556 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:40.406735 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:40.406623 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:16:40.406735 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:40.406690 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:41.406674521 +0000 UTC m=+235.001673263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : configmap "kube-rbac-proxy" not found Sep 08 16:16:40.406735 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:40.406707 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:16:40.406890 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:40.406747 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:41.406736549 +0000 UTC m=+235.001735285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : secret "insights-runtime-extractor-tls" not found Sep 08 16:16:40.644800 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:40.644767 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" event={"ID":"0cd2d9b3-c508-4471-b728-d5faf5bcf5b1","Type":"ContainerStarted","Data":"b6819c6aeaa633819f77c98d388747aa0a8fce0e8ba36c64ee8e94b27307c5fc"} Sep 08 16:16:41.416266 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:41.416190 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:41.416466 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:41.416280 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:41.416466 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:41.416345 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:16:41.416466 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:41.416426 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:16:41.416466 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:41.416435 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:43.416411805 +0000 UTC m=+237.011410558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : configmap "kube-rbac-proxy" not found Sep 08 16:16:41.416639 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:41.416481 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:43.416464685 +0000 UTC m=+237.011463422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : secret "insights-runtime-extractor-tls" not found Sep 08 16:16:42.423681 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:42.423639 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:42.423681 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:42.423688 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:42.424239 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:42.423722 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:42.424239 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.423808 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:42.424239 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.423822 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 16:16:42.424239 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.423815 2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 08 16:16:42.424239 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.423872 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls podName:b979f2af-e275-4d96-a9aa-4c5d72a1410e nodeName:}" failed. No retries permitted until 2026-09-08 16:16:50.423855742 +0000 UTC m=+244.018854483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vk2j" (UID: "b979f2af-e275-4d96-a9aa-4c5d72a1410e") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:42.424239 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.423886 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert podName:aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa nodeName:}" failed. No retries permitted until 2026-09-08 16:16:50.423879566 +0000 UTC m=+244.018878302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f9dn5" (UID: "aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa") : secret "networking-console-plugin-cert" not found Sep 08 16:16:42.424239 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.423898 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls podName:492953d2-fe38-47ba-aec4-d6a8a70c4dc5 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:50.42389192 +0000 UTC m=+244.018890657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6fbgb" (UID: "492953d2-fe38-47ba-aec4-d6a8a70c4dc5") : secret "samples-operator-tls" not found Sep 08 16:16:42.524617 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:42.524572 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:42.524797 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.524721 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 16:16:42.524797 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.524751 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-844f884987-42b4w: secret "image-registry-tls" not found Sep 08 16:16:42.524882 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:42.524807 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls podName:b3c44751-cd04-4aa8-9bdf-3cf9dad048d2 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:50.524791498 +0000 UTC m=+244.119790236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls") pod "image-registry-844f884987-42b4w" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2") : secret "image-registry-tls" not found Sep 08 16:16:42.564633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:42.564606 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxdnw_f0ff108a-6700-47c6-b566-f6e6740810e1/dns-node-resolver/0.log" Sep 08 16:16:43.433161 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:43.433126 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:43.433518 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:43.433172 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:43.433518 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:43.433267 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:16:43.433518 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:43.433290 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:16:43.433518 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:43.433345 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:47.433325488 +0000 UTC m=+241.028324225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : configmap "kube-rbac-proxy" not found Sep 08 16:16:43.433518 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:43.433362 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:47.433355561 +0000 UTC m=+241.028354298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : secret "insights-runtime-extractor-tls" not found Sep 08 16:16:43.569446 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:43.569420 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nsbsw_02929572-64d2-492c-8602-3a3dc9bc9701/node-ca/0.log" Sep 08 16:16:43.656349 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:43.656308 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" event={"ID":"0cd2d9b3-c508-4471-b728-d5faf5bcf5b1","Type":"ContainerStarted","Data":"65075a1f49b4356355b0658466f45b1886d6b201735957c73fb91fcff650695d"} Sep 08 16:16:43.656529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:43.656373 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" event={"ID":"0cd2d9b3-c508-4471-b728-d5faf5bcf5b1","Type":"ContainerStarted","Data":"beabbbe24031cea135f47ac8efe148dd3ace5a31768e503b0016497682926a83"} Sep 08 16:16:43.675327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:43.675273 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5mj92" podStartSLOduration=1.542090776 podStartE2EDuration="4.675256173s" totalImagesPullingTime="3.133165397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:43.67346807 +0000 UTC m=+237.268466830" watchObservedRunningTime="2026-09-08 16:16:43.675256173 +0000 UTC m=+237.270254923" Sep 08 16:16:44.165830 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:44.165796 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-765jc_b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008/serve-healthcheck-canary/0.log" Sep 08 16:16:44.766320 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:44.766286 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5mj92_0cd2d9b3-c508-4471-b728-d5faf5bcf5b1/migrator/0.log" Sep 08 16:16:44.965513 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:44.965428 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5mj92_0cd2d9b3-c508-4471-b728-d5faf5bcf5b1/graceful-termination/0.log" Sep 08 16:16:45.166651 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:45.166615 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-65p48_1e9c00eb-2239-4e76-9155-8bb822586051/kube-storage-version-migrator-operator/0.log" Sep 08 16:16:47.465941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:47.465905 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:47.466345 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:47.465952 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:47.466345 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:47.466030 2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 16:16:47.466345 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:47.466044 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:16:47.466345 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:47.466102 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:55.46608077 +0000 UTC m=+249.061079512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : configmap "kube-rbac-proxy" not found Sep 08 16:16:47.466345 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:47.466118 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:55.466111925 +0000 UTC m=+249.061110661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : secret "insights-runtime-extractor-tls" not found Sep 08 16:16:50.491656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.491610 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:16:50.492055 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.491751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:50.492055 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:50.491782 2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 16:16:50.492055 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:50.491850 2555 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:50.492055 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:50.491854 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert podName:aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa nodeName:}" failed. No retries permitted until 2026-09-08 16:17:06.491833567 +0000 UTC m=+260.086832327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-f9dn5" (UID: "aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa") : secret "networking-console-plugin-cert" not found Sep 08 16:16:50.492055 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.491787 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:16:50.492055 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:50.491904 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls podName:b979f2af-e275-4d96-a9aa-4c5d72a1410e nodeName:}" failed. No retries permitted until 2026-09-08 16:17:06.49189296 +0000 UTC m=+260.086891701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vk2j" (UID: "b979f2af-e275-4d96-a9aa-4c5d72a1410e") : secret "cluster-monitoring-operator-tls" not found Sep 08 16:16:50.494231 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.494198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492953d2-fe38-47ba-aec4-d6a8a70c4dc5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6fbgb\" (UID: \"492953d2-fe38-47ba-aec4-d6a8a70c4dc5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:50.593297 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.593255 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:50.595706 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.595677 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"image-registry-844f884987-42b4w\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:50.638776 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.638745 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-85mxb\"" Sep 08 16:16:50.644119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.644097 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgmfx\"" Sep 08 16:16:50.647019 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.647003 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" Sep 08 16:16:50.651988 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.651972 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:50.918464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.918427 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb"] Sep 08 16:16:50.935299 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:50.935278 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-844f884987-42b4w"] Sep 08 16:16:50.937318 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:16:50.937287 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c44751_cd04_4aa8_9bdf_3cf9dad048d2.slice/crio-e6952b0b73e2e9ce43fb47bc4a9509f40867db19e80f85b01534dd07108d97ce WatchSource:0}: Error finding container e6952b0b73e2e9ce43fb47bc4a9509f40867db19e80f85b01534dd07108d97ce: Status 404 returned error can't find the container with id e6952b0b73e2e9ce43fb47bc4a9509f40867db19e80f85b01534dd07108d97ce Sep 08 16:16:51.683620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:51.683583 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-844f884987-42b4w" event={"ID":"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2","Type":"ContainerStarted","Data":"ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a"} Sep 08 16:16:51.684064 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:51.683633 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-844f884987-42b4w" event={"ID":"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2","Type":"ContainerStarted","Data":"e6952b0b73e2e9ce43fb47bc4a9509f40867db19e80f85b01534dd07108d97ce"} Sep 08 16:16:51.684064 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:51.683741 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:16:51.684733 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:51.684713 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" event={"ID":"492953d2-fe38-47ba-aec4-d6a8a70c4dc5","Type":"ContainerStarted","Data":"e438a11d2d2aa9495bf74d1ba62e59ffa4cdbe6e23e8536b2be3d39befa1e1d5"} Sep 08 16:16:51.704087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:51.704024 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-844f884987-42b4w" podStartSLOduration=17.704001106 podStartE2EDuration="17.704001106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:51.702529619 +0000 UTC m=+245.297528403" watchObservedRunningTime="2026-09-08 16:16:51.704001106 +0000 UTC m=+245.298999869" Sep 08 16:16:53.693623 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:53.693583 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" event={"ID":"492953d2-fe38-47ba-aec4-d6a8a70c4dc5","Type":"ContainerStarted","Data":"442b0a9887dc63c44f2d96fdc5c048b136f113dbe1d61279138bae048102f531"} Sep 08 16:16:53.694035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:53.693647 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" event={"ID":"492953d2-fe38-47ba-aec4-d6a8a70c4dc5","Type":"ContainerStarted","Data":"ec5f888f05897ed2381a83e0d76a736d799092e57b4d71c678cde7293ff61dee"} Sep 08 16:16:53.712015 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:53.711954 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6fbgb" podStartSLOduration=17.752611598 podStartE2EDuration="19.71193763s" totalImagesPullingTime="1.959326032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:53.710837811 +0000 UTC m=+247.305836587" watchObservedRunningTime="2026-09-08 16:16:53.71193763 +0000 UTC m=+247.306936389" Sep 08 16:16:55.536556 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:55.536506 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:55.536556 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:55.536560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:55.537105 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:55.536668 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 16:16:55.537105 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:16:55.536721 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls podName:f4bed502-a9da-4bcb-a714-256171637ea8 nodeName:}" failed. No retries permitted until 2026-09-08 16:17:11.536707196 +0000 UTC m=+265.131705932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d5px9" (UID: "f4bed502-a9da-4bcb-a714-256171637ea8") : secret "insights-runtime-extractor-tls" not found Sep 08 16:16:55.537105 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:55.537004 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f4bed502-a9da-4bcb-a714-256171637ea8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:16:58.660657 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:58.660605 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:16:58.662905 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:58.662884 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fb613c0-0100-414c-b958-e753b2d9a521-metrics-certs\") pod \"network-metrics-daemon-fbd6c\" (UID: \"3fb613c0-0100-414c-b958-e753b2d9a521\") " pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:16:58.835404 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:58.835373 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rq8v\"" Sep 08 16:16:58.843428 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:58.843399 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbd6c" Sep 08 16:16:59.111339 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:59.111307 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fbd6c"] Sep 08 16:16:59.114259 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:16:59.114214 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb613c0_0100_414c_b958_e753b2d9a521.slice/crio-3de8b9ffd75cbd71cceeae6305859d6f3032701b5b891d499518a2e067797081 WatchSource:0}: Error finding container 3de8b9ffd75cbd71cceeae6305859d6f3032701b5b891d499518a2e067797081: Status 404 returned error can't find the container with id 3de8b9ffd75cbd71cceeae6305859d6f3032701b5b891d499518a2e067797081 Sep 08 16:16:59.715087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:16:59.715042 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fbd6c" event={"ID":"3fb613c0-0100-414c-b958-e753b2d9a521","Type":"ContainerStarted","Data":"3de8b9ffd75cbd71cceeae6305859d6f3032701b5b891d499518a2e067797081"} Sep 08 16:17:00.720150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:00.720116 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fbd6c" event={"ID":"3fb613c0-0100-414c-b958-e753b2d9a521","Type":"ContainerStarted","Data":"9c5a176d79f45d1f5bd61ae0335ba344df526a7ac55fbddda643f1ede20c68b5"} Sep 08 16:17:00.720573 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:00.720168 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fbd6c" event={"ID":"3fb613c0-0100-414c-b958-e753b2d9a521","Type":"ContainerStarted","Data":"317f0bfaeccb40fbf32ab2f59683c48f13de310aafc68deb8bec971a9deb1d53"} Sep 08 16:17:00.736292 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:00.736209 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fbd6c" podStartSLOduration=252.724988846 podStartE2EDuration="4m13.736191191s" totalImagesPullingTime="1.011202345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:00.735588238 +0000 UTC m=+254.330587007" watchObservedRunningTime="2026-09-08 16:17:00.736191191 +0000 UTC m=+254.331189951" Sep 08 16:17:05.498565 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:05.498526 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-844f884987-42b4w"] Sep 08 16:17:06.524600 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.524547 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:17:06.524600 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.524611 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:17:06.527069 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.527041 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b979f2af-e275-4d96-a9aa-4c5d72a1410e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vk2j\" (UID: \"b979f2af-e275-4d96-a9aa-4c5d72a1410e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:17:06.527145 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.527110 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-f9dn5\" (UID: \"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa\") " pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:17:06.766852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.766820 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wb4mg\"" Sep 08 16:17:06.774934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.774863 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" Sep 08 16:17:06.824899 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.824876 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cmc99\"" Sep 08 16:17:06.833102 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:06.833069 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" Sep 08 16:17:07.045649 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:07.045567 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j"] Sep 08 16:17:07.048968 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:17:07.048934 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb979f2af_e275_4d96_a9aa_4c5d72a1410e.slice/crio-5336dda2ac6dcb017ceeb437ec4353c543e68f5059a7a22c31f1d17712414b21 WatchSource:0}: Error finding container 5336dda2ac6dcb017ceeb437ec4353c543e68f5059a7a22c31f1d17712414b21: Status 404 returned error can't find the container with id 5336dda2ac6dcb017ceeb437ec4353c543e68f5059a7a22c31f1d17712414b21 Sep 08 16:17:07.104925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:07.104880 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-f9dn5"] Sep 08 16:17:07.107555 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:17:07.107526 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae8b909_2a17_49e5_ad81_d0ce8cf3fcfa.slice/crio-3068807af28d1678edfd3bb665c4768c613d0731f8dcedf2e5f3e91fa8b2537c WatchSource:0}: Error finding container 3068807af28d1678edfd3bb665c4768c613d0731f8dcedf2e5f3e91fa8b2537c: Status 404 returned error can't find the container with id 3068807af28d1678edfd3bb665c4768c613d0731f8dcedf2e5f3e91fa8b2537c Sep 08 16:17:07.744871 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:07.744834 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" event={"ID":"b979f2af-e275-4d96-a9aa-4c5d72a1410e","Type":"ContainerStarted","Data":"5336dda2ac6dcb017ceeb437ec4353c543e68f5059a7a22c31f1d17712414b21"} Sep 08 16:17:07.745962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:07.745936 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" event={"ID":"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa","Type":"ContainerStarted","Data":"3068807af28d1678edfd3bb665c4768c613d0731f8dcedf2e5f3e91fa8b2537c"} Sep 08 16:17:09.754195 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:09.754156 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" event={"ID":"b979f2af-e275-4d96-a9aa-4c5d72a1410e","Type":"ContainerStarted","Data":"3c34357e17ab9d194ede90ae87ff17d5f37ac658c31da661634a0c67ac8a0c45"} Sep 08 16:17:09.755651 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:09.755627 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" event={"ID":"aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa","Type":"ContainerStarted","Data":"9cfdf12b810a0b1d3d6d6e17a8051ea0520329f83afd8fbb0a295b42e5b07c61"} Sep 08 16:17:09.771814 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:09.771756 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vk2j" podStartSLOduration=33.90584705 podStartE2EDuration="35.771738284s" totalImagesPullingTime="1.865891234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:09.770122099 +0000 UTC m=+263.365120858" watchObservedRunningTime="2026-09-08 16:17:09.771738284 +0000 UTC m=+263.366737043" Sep 08 16:17:09.785986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:09.785926 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-f9dn5" podStartSLOduration=34.473680022 podStartE2EDuration="35.785909181s" totalImagesPullingTime="1.312229159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:09.784732301 +0000 UTC m=+263.379731076" watchObservedRunningTime="2026-09-08 16:17:09.785909181 +0000 UTC m=+263.380907940" Sep 08 16:17:11.564744 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:11.564687 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:17:11.566954 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:11.566934 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f4bed502-a9da-4bcb-a714-256171637ea8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d5px9\" (UID: \"f4bed502-a9da-4bcb-a714-256171637ea8\") " pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:17:11.809854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:11.809818 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mlgpx\"" Sep 08 16:17:11.817805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:11.817731 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d5px9" Sep 08 16:17:12.085510 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:12.085424 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d5px9"] Sep 08 16:17:12.088157 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:17:12.088119 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bed502_a9da_4bcb_a714_256171637ea8.slice/crio-343d714b5ca5d57003ad6251b452aca9d5e8a16d3ecc3bc64c582ffc3f04522c WatchSource:0}: Error finding container 343d714b5ca5d57003ad6251b452aca9d5e8a16d3ecc3bc64c582ffc3f04522c: Status 404 returned error can't find the container with id 343d714b5ca5d57003ad6251b452aca9d5e8a16d3ecc3bc64c582ffc3f04522c Sep 08 16:17:12.766443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:12.766400 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d5px9" event={"ID":"f4bed502-a9da-4bcb-a714-256171637ea8","Type":"ContainerStarted","Data":"b4a2e87dd817206a30717fbed9eb74de8bff12f59ddc5b806c89043d0b876011"} Sep 08 16:17:12.766847 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:12.766489 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d5px9" event={"ID":"f4bed502-a9da-4bcb-a714-256171637ea8","Type":"ContainerStarted","Data":"343d714b5ca5d57003ad6251b452aca9d5e8a16d3ecc3bc64c582ffc3f04522c"} Sep 08 16:17:13.771722 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:13.771688 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d5px9" event={"ID":"f4bed502-a9da-4bcb-a714-256171637ea8","Type":"ContainerStarted","Data":"e857b3803d7284343333a2b37a2e29de9ccd12e0bf8be027f257e2f46a8de251"} Sep 08 16:17:15.504659 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:15.504595 2555 patch_prober.go:28] interesting pod/image-registry-844f884987-42b4w container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 08 16:17:15.505072 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:15.504669 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-844f884987-42b4w" podUID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 16:17:16.783609 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.783572 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d5px9" event={"ID":"f4bed502-a9da-4bcb-a714-256171637ea8","Type":"ContainerStarted","Data":"326fbfb478a103aaa70acb405ac5f57bef4c06bd5abc4edb76ebfa015e987f0e"} Sep 08 16:17:16.804907 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.804872 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr"] Sep 08 16:17:16.807737 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.807718 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:16.810198 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.810154 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d5px9" podStartSLOduration=33.909998977 podStartE2EDuration="37.810138537s" totalImagesPullingTime="3.90013956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:16:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:16.809188909 +0000 UTC m=+270.404187686" watchObservedRunningTime="2026-09-08 16:17:16.810138537 +0000 UTC m=+270.405137295" Sep 08 16:17:16.810709 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.810676 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 08 16:17:16.810830 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.810711 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 16:17:16.810830 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.810715 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 08 16:17:16.811200 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.811181 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dr7dt\"" Sep 08 16:17:16.820573 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.820547 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pbfjk"] Sep 08 16:17:16.822833 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.822812 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.824413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.824390 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr"] Sep 08 16:17:16.825507 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.825484 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dqlk\"" Sep 08 16:17:16.825613 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.825484 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 16:17:16.825725 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.825710 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 16:17:16.825793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.825775 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 16:17:16.904353 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904314 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:16.904353 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904354 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:16.904627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904451 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-tls\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904496 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zc8\" (UniqueName: \"kubernetes.io/projected/fc6e78d5-196b-493b-bd2a-fdecf328afce-kube-api-access-q2zc8\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904536 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-textfile\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904558 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e78d5-196b-493b-bd2a-fdecf328afce-metrics-client-ca\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904581 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904609 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-sys\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904659 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-root\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904699 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkt7\" (UniqueName: \"kubernetes.io/projected/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-kube-api-access-gfkt7\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:16.904854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904722 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-wtmp\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904746 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:16.904854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:16.904764 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.006001 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.005954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkt7\" (UniqueName: \"kubernetes.io/projected/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-kube-api-access-gfkt7\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.006001 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006002 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-wtmp\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006034 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006060 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.006294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006094 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.006294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006126 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.006294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006182 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-wtmp\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006312 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-tls\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006358 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zc8\" (UniqueName: \"kubernetes.io/projected/fc6e78d5-196b-493b-bd2a-fdecf328afce-kube-api-access-q2zc8\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006399 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-textfile\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006423 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e78d5-196b-493b-bd2a-fdecf328afce-metrics-client-ca\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006448 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006477 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-sys\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006506 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-root\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006846 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006581 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-root\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006846 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006800 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-accelerators-collector-config\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006949 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006859 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc6e78d5-196b-493b-bd2a-fdecf328afce-sys\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.006949 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.006913 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.007083 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.007064 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc6e78d5-196b-493b-bd2a-fdecf328afce-metrics-client-ca\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.007235 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.007214 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-textfile\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.008825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.008805 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.008924 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.008864 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.009112 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.009092 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc6e78d5-196b-493b-bd2a-fdecf328afce-node-exporter-tls\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.009177 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.009161 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.014794 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.014769 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkt7\" (UniqueName: \"kubernetes.io/projected/1ba98751-9c94-4a95-8c0b-ef0f505acc6e-kube-api-access-gfkt7\") pod \"openshift-state-metrics-657fd6c54-dwplr\" (UID: \"1ba98751-9c94-4a95-8c0b-ef0f505acc6e\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.015304 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.015281 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zc8\" (UniqueName: \"kubernetes.io/projected/fc6e78d5-196b-493b-bd2a-fdecf328afce-kube-api-access-q2zc8\") pod \"node-exporter-pbfjk\" (UID: \"fc6e78d5-196b-493b-bd2a-fdecf328afce\") " pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.117047 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.117003 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" Sep 08 16:17:17.133268 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.133211 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pbfjk" Sep 08 16:17:17.144366 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:17:17.144311 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6e78d5_196b_493b_bd2a_fdecf328afce.slice/crio-c0b2f78d8e069ba0eb75c85f72bca3fe5d4b9d3d5a80140d55f4130e3eb54ea2 WatchSource:0}: Error finding container c0b2f78d8e069ba0eb75c85f72bca3fe5d4b9d3d5a80140d55f4130e3eb54ea2: Status 404 returned error can't find the container with id c0b2f78d8e069ba0eb75c85f72bca3fe5d4b9d3d5a80140d55f4130e3eb54ea2 Sep 08 16:17:17.389920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.389708 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr"] Sep 08 16:17:17.394968 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:17:17.394941 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba98751_9c94_4a95_8c0b_ef0f505acc6e.slice/crio-0909538cc165c3e00b4544fb327b47238af59edf0617377524d49ed95fe0216f WatchSource:0}: Error finding container 0909538cc165c3e00b4544fb327b47238af59edf0617377524d49ed95fe0216f: Status 404 returned error can't find the container with id 0909538cc165c3e00b4544fb327b47238af59edf0617377524d49ed95fe0216f Sep 08 16:17:17.789351 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.789231 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbfjk" event={"ID":"fc6e78d5-196b-493b-bd2a-fdecf328afce","Type":"ContainerStarted","Data":"c0b2f78d8e069ba0eb75c85f72bca3fe5d4b9d3d5a80140d55f4130e3eb54ea2"} Sep 08 16:17:17.791272 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.791228 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" event={"ID":"1ba98751-9c94-4a95-8c0b-ef0f505acc6e","Type":"ContainerStarted","Data":"9199b5056ce51ffe8388a18b399a3f738bba7efdeac93ab8d8419c71fd39c467"} Sep 08 16:17:17.791401 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.791299 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" event={"ID":"1ba98751-9c94-4a95-8c0b-ef0f505acc6e","Type":"ContainerStarted","Data":"9ca3b11e33b406ddcf236f5b07c31709818a9e8d4f6863081839c70a3230edab"} Sep 08 16:17:17.791401 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:17.791319 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" event={"ID":"1ba98751-9c94-4a95-8c0b-ef0f505acc6e","Type":"ContainerStarted","Data":"0909538cc165c3e00b4544fb327b47238af59edf0617377524d49ed95fe0216f"} Sep 08 16:17:18.797400 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:18.797309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" event={"ID":"1ba98751-9c94-4a95-8c0b-ef0f505acc6e","Type":"ContainerStarted","Data":"b5a180261aeed9236daedf6fc0ae7bbc11603e614455e7787c500070511225c9"} Sep 08 16:17:18.815132 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:18.815077 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-dwplr" podStartSLOduration=1.790515095 podStartE2EDuration="2.815061218s" totalImagesPullingTime="1.024546123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:18.813010838 +0000 UTC m=+272.408009612" watchObservedRunningTime="2026-09-08 16:17:18.815061218 +0000 UTC m=+272.410059977" Sep 08 16:17:19.803115 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:19.803018 2555 generic.go:356] "Generic (PLEG): container finished" podID="fc6e78d5-196b-493b-bd2a-fdecf328afce" containerID="3ecc5bceab809ace1d7fd0a626586e2c522e7d542e6c9d2e3167f1e4d723257f" exitCode=0 Sep 08 16:17:19.803560 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:19.803106 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbfjk" event={"ID":"fc6e78d5-196b-493b-bd2a-fdecf328afce","Type":"ContainerDied","Data":"3ecc5bceab809ace1d7fd0a626586e2c522e7d542e6c9d2e3167f1e4d723257f"} Sep 08 16:17:20.808873 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:20.808839 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbfjk" event={"ID":"fc6e78d5-196b-493b-bd2a-fdecf328afce","Type":"ContainerStarted","Data":"06468fc5e22a1c5300d4d2d50e5c3cd8d482ca52870deafcda54c04678823372"} Sep 08 16:17:20.809353 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:20.808890 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pbfjk" event={"ID":"fc6e78d5-196b-493b-bd2a-fdecf328afce","Type":"ContainerStarted","Data":"0684642854c8061b94c490a0c2486f0a514b1254f9ffe2200a1c1886e2c3e658"} Sep 08 16:17:20.829596 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:20.829540 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pbfjk" podStartSLOduration=2.393406373 podStartE2EDuration="4.829522247s" totalImagesPullingTime="2.369805833s" totalInitContainerRuntime="66.310041ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:20.827885782 +0000 UTC m=+274.422884566" watchObservedRunningTime="2026-09-08 16:17:20.829522247 +0000 UTC m=+274.424521006" Sep 08 16:17:25.503616 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:25.503576 2555 patch_prober.go:28] interesting pod/image-registry-844f884987-42b4w container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 08 16:17:25.503996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:25.503637 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-844f884987-42b4w" podUID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 16:17:28.437617 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:17:28.437568 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-j4tfc" podUID="a5881b84-4529-469e-ac2e-9ff4ca8726ef" Sep 08 16:17:28.836313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:28.836205 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j4tfc" Sep 08 16:17:30.517999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.517933 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-844f884987-42b4w" podUID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" containerName="registry" containerID="cri-o://ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a" gracePeriod=30 Sep 08 16:17:30.757226 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.757199 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:17:30.820086 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.819992 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-installation-pull-secrets\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820086 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820034 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-certificates\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820086 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820077 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcgbl\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-kube-api-access-wcgbl\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820155 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-ca-trust-extracted\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820208 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-trusted-ca\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820274 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-bound-sa-token\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820318 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-image-registry-private-configuration\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820348 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") pod \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\" (UID: \"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2\") " Sep 08 16:17:30.820654 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.820577 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:17:30.821113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.821077 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:17:30.822795 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.822715 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:17:30.822893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.822827 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:17:30.822893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.822831 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:17:30.823017 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.822897 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:17:30.823017 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.822924 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-kube-api-access-wcgbl" (OuterVolumeSpecName: "kube-api-access-wcgbl") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "kube-api-access-wcgbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:17:30.828554 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.828523 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" (UID: "b3c44751-cd04-4aa8-9bdf-3cf9dad048d2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:17:30.844423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.844391 2555 generic.go:356] "Generic (PLEG): container finished" podID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" containerID="ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a" exitCode=0 Sep 08 16:17:30.844585 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.844435 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-844f884987-42b4w" Sep 08 16:17:30.844585 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.844476 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-844f884987-42b4w" event={"ID":"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2","Type":"ContainerDied","Data":"ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a"} Sep 08 16:17:30.844585 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.844527 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-844f884987-42b4w" event={"ID":"b3c44751-cd04-4aa8-9bdf-3cf9dad048d2","Type":"ContainerDied","Data":"e6952b0b73e2e9ce43fb47bc4a9509f40867db19e80f85b01534dd07108d97ce"} Sep 08 16:17:30.845194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.845178 2555 scope.go:117] "RemoveContainer" containerID="ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a" Sep 08 16:17:30.853631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.853615 2555 scope.go:117] "RemoveContainer" containerID="ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a" Sep 08 16:17:30.853942 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:17:30.853919 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a\": container with ID starting with ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a not found: ID does not exist" containerID="ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a" Sep 08 16:17:30.854040 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.853950 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a"} err="failed to get container status \"ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a\": rpc error: code = NotFound desc = could not find container \"ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a\": container with ID starting with ff712942c781b774de3829b51feb7a30aee1407d0f63662f610f6e2f068b511a not found: ID does not exist" Sep 08 16:17:30.864794 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.864763 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-844f884987-42b4w"] Sep 08 16:17:30.868003 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.867979 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-844f884987-42b4w"] Sep 08 16:17:30.921797 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921762 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-installation-pull-secrets\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.921797 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921789 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-certificates\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.921797 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921801 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcgbl\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-kube-api-access-wcgbl\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.921797 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921809 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-ca-trust-extracted\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.922142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921818 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-trusted-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.922142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921826 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-bound-sa-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.922142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921834 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-image-registry-private-configuration\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.922142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.921845 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2-registry-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:17:30.935625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:30.935596 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" path="/var/lib/kubelet/pods/b3c44751-cd04-4aa8-9bdf-3cf9dad048d2/volumes" Sep 08 16:17:31.728620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:31.728556 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:17:31.730842 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:31.730816 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5881b84-4529-469e-ac2e-9ff4ca8726ef-metrics-tls\") pod \"dns-default-j4tfc\" (UID: \"a5881b84-4529-469e-ac2e-9ff4ca8726ef\") " pod="openshift-dns/dns-default-j4tfc" Sep 08 16:17:31.839600 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:31.839557 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlhzx\"" Sep 08 16:17:31.847644 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:31.847612 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j4tfc" Sep 08 16:17:32.115739 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:32.115712 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j4tfc"] Sep 08 16:17:32.118417 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:17:32.118391 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5881b84_4529_469e_ac2e_9ff4ca8726ef.slice/crio-83dd5d69870485da31c83912c84872f1a7fdb4fd01430f1dc05e4297db9acaa3 WatchSource:0}: Error finding container 83dd5d69870485da31c83912c84872f1a7fdb4fd01430f1dc05e4297db9acaa3: Status 404 returned error can't find the container with id 83dd5d69870485da31c83912c84872f1a7fdb4fd01430f1dc05e4297db9acaa3 Sep 08 16:17:32.855659 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:32.855623 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j4tfc" event={"ID":"a5881b84-4529-469e-ac2e-9ff4ca8726ef","Type":"ContainerStarted","Data":"83dd5d69870485da31c83912c84872f1a7fdb4fd01430f1dc05e4297db9acaa3"} Sep 08 16:17:33.862364 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:33.862271 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j4tfc" event={"ID":"a5881b84-4529-469e-ac2e-9ff4ca8726ef","Type":"ContainerStarted","Data":"2711fe1db15d4472da88bda1004b99101c4cce4c1647d0bf70528e5beb0e5b23"} Sep 08 16:17:33.862754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:33.862379 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j4tfc" event={"ID":"a5881b84-4529-469e-ac2e-9ff4ca8726ef","Type":"ContainerStarted","Data":"5379d2f13411a0b6d73b0b6ddf0bec5acf4de4e7edd60f7715364f39f811c99d"} Sep 08 16:17:33.862754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:33.862412 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j4tfc" Sep 08 16:17:33.882333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:33.882279 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j4tfc" podStartSLOduration=251.397932822 podStartE2EDuration="4m12.882240037s" totalImagesPullingTime="1.484307215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:33.880741493 +0000 UTC m=+287.475740279" watchObservedRunningTime="2026-09-08 16:17:33.882240037 +0000 UTC m=+287.477238796" Sep 08 16:17:37.697431 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.697389 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6598cc7fc5-g5j7f"] Sep 08 16:17:37.697838 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.697690 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" containerName="registry" Sep 08 16:17:37.697838 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.697701 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" containerName="registry" Sep 08 16:17:37.697838 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.697763 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c44751-cd04-4aa8-9bdf-3cf9dad048d2" containerName="registry" Sep 08 16:17:37.699998 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.699980 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.702266 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.702224 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 16:17:37.702409 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.702333 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 16:17:37.702409 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.702334 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dz7wm\"" Sep 08 16:17:37.703160 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.703141 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 16:17:37.703286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.703193 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 16:17:37.703286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.703223 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 16:17:37.703464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.703449 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 16:17:37.703525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.703470 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 16:17:37.707896 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.707875 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 16:17:37.710170 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.710151 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6598cc7fc5-g5j7f"] Sep 08 16:17:37.771181 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.771143 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-service-ca\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.771181 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.771183 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4x2\" (UniqueName: \"kubernetes.io/projected/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-kube-api-access-vp4x2\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.771406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.771202 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-trusted-ca-bundle\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.771406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.771222 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-oauth-config\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.771406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.771258 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-oauth-serving-cert\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.771406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.771309 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-config\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.771406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.771332 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-serving-cert\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.872540 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.872487 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-service-ca\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.872540 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.872543 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4x2\" (UniqueName: \"kubernetes.io/projected/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-kube-api-access-vp4x2\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.872799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.872560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-trusted-ca-bundle\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.872799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.872580 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-oauth-config\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.872799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.872606 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-oauth-serving-cert\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.872799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.872646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-config\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.872799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.872675 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-serving-cert\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.873387 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.873361 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-service-ca\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.873586 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.873564 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-config\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.873652 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.873639 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-trusted-ca-bundle\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.874049 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.874027 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-oauth-serving-cert\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.875233 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.875212 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-oauth-config\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.875436 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.875416 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-serving-cert\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:37.882297 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:37.882274 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4x2\" (UniqueName: \"kubernetes.io/projected/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-kube-api-access-vp4x2\") pod \"console-6598cc7fc5-g5j7f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:38.010725 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:38.010625 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:38.280826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:38.280800 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6598cc7fc5-g5j7f"] Sep 08 16:17:38.283034 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:17:38.283006 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c3200d_83e3_45e1_be2c_2395ff6d7b3f.slice/crio-644d8e487245492ea3af01393b6ea77fe2997ee547353c96b9dfa91a448bc906 WatchSource:0}: Error finding container 644d8e487245492ea3af01393b6ea77fe2997ee547353c96b9dfa91a448bc906: Status 404 returned error can't find the container with id 644d8e487245492ea3af01393b6ea77fe2997ee547353c96b9dfa91a448bc906 Sep 08 16:17:38.887574 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:38.887539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6598cc7fc5-g5j7f" event={"ID":"98c3200d-83e3-45e1-be2c-2395ff6d7b3f","Type":"ContainerStarted","Data":"644d8e487245492ea3af01393b6ea77fe2997ee547353c96b9dfa91a448bc906"} Sep 08 16:17:41.900098 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:41.900060 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6598cc7fc5-g5j7f" event={"ID":"98c3200d-83e3-45e1-be2c-2395ff6d7b3f","Type":"ContainerStarted","Data":"5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897"} Sep 08 16:17:41.917197 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:41.917134 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6598cc7fc5-g5j7f" podStartSLOduration=2.223948454 podStartE2EDuration="4.917118773s" totalImagesPullingTime="2.693170319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:41.916420278 +0000 UTC m=+295.511419067" watchObservedRunningTime="2026-09-08 16:17:41.917118773 +0000 UTC m=+295.512117531" Sep 08 16:17:43.868838 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:43.868806 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j4tfc" Sep 08 16:17:46.786018 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:46.785987 2555 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 16:17:48.011773 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:48.011728 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:48.014038 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:48.011796 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:48.016568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:48.016543 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:48.934961 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:48.934927 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:17:51.940223 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:51.940135 2555 generic.go:356] "Generic (PLEG): container finished" podID="4f36e387-c3ae-47c5-81cd-ace60aca2f38" containerID="80fd9b3925d85a57060f0acebb9d15be7332650c1c736d3f577b7a9424c0d9ec" exitCode=255 Sep 08 16:17:51.940223 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:51.940207 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" event={"ID":"4f36e387-c3ae-47c5-81cd-ace60aca2f38","Type":"ContainerDied","Data":"80fd9b3925d85a57060f0acebb9d15be7332650c1c736d3f577b7a9424c0d9ec"} Sep 08 16:17:51.940700 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:51.940276 2555 scope.go:117] "RemoveContainer" containerID="d7a4d516df2d42514faca5a5de041ef481d78f7b2a027729b9b471aec0f0a8b7" Sep 08 16:17:51.946530 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:51.946510 2555 scope.go:117] "RemoveContainer" containerID="80fd9b3925d85a57060f0acebb9d15be7332650c1c736d3f577b7a9424c0d9ec" Sep 08 16:17:51.946707 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:17:51.946690 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-694cc5f4dd-czffm_open-cluster-management-agent-addon(4f36e387-c3ae-47c5-81cd-ace60aca2f38)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" podUID="4f36e387-c3ae-47c5-81cd-ace60aca2f38" Sep 08 16:17:57.418229 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:57.418177 2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" Sep 08 16:17:57.418699 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:17:57.418564 2555 scope.go:117] "RemoveContainer" containerID="80fd9b3925d85a57060f0acebb9d15be7332650c1c736d3f577b7a9424c0d9ec" Sep 08 16:17:57.418759 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:17:57.418739 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-694cc5f4dd-czffm_open-cluster-management-agent-addon(4f36e387-c3ae-47c5-81cd-ace60aca2f38)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" podUID="4f36e387-c3ae-47c5-81cd-ace60aca2f38" Sep 08 16:18:04.990045 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:04.990007 2555 generic.go:356] "Generic (PLEG): container finished" podID="ee82cb26-8982-49a1-9d19-03981de45ab9" containerID="06f4c5c120d0bff8a62b84a6901967c722038f6a6edddea7d5821b7bcffbcaa4" exitCode=0 Sep 08 16:18:04.990454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:04.990081 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" event={"ID":"ee82cb26-8982-49a1-9d19-03981de45ab9","Type":"ContainerDied","Data":"06f4c5c120d0bff8a62b84a6901967c722038f6a6edddea7d5821b7bcffbcaa4"} Sep 08 16:18:04.990454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:04.990418 2555 scope.go:117] "RemoveContainer" containerID="06f4c5c120d0bff8a62b84a6901967c722038f6a6edddea7d5821b7bcffbcaa4" Sep 08 16:18:05.995137 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:05.995103 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-clpvs" event={"ID":"ee82cb26-8982-49a1-9d19-03981de45ab9","Type":"ContainerStarted","Data":"3de8a6104f46e2cc298522e2a90409d764c91ffde588f9b4ce581d3f9b0d4617"} Sep 08 16:18:08.931735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:08.931690 2555 scope.go:117] "RemoveContainer" containerID="80fd9b3925d85a57060f0acebb9d15be7332650c1c736d3f577b7a9424c0d9ec" Sep 08 16:18:09.007771 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:09.007742 2555 generic.go:356] "Generic (PLEG): container finished" podID="1e9c00eb-2239-4e76-9155-8bb822586051" containerID="e9b75dffcc7a789ae94c6568b1a1ad39b73a1f27256650d04794d8d68f7676e0" exitCode=0 Sep 08 16:18:09.007887 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:09.007809 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" event={"ID":"1e9c00eb-2239-4e76-9155-8bb822586051","Type":"ContainerDied","Data":"e9b75dffcc7a789ae94c6568b1a1ad39b73a1f27256650d04794d8d68f7676e0"} Sep 08 16:18:09.008161 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:09.008143 2555 scope.go:117] "RemoveContainer" containerID="e9b75dffcc7a789ae94c6568b1a1ad39b73a1f27256650d04794d8d68f7676e0" Sep 08 16:18:09.009392 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:09.009367 2555 generic.go:356] "Generic (PLEG): container finished" podID="a1a5a8ea-d576-42a5-a612-fd894d234b40" containerID="db80ae435ced91d45c81702077ffa60d9aec767d0a884f4474add711d50af52e" exitCode=0 Sep 08 16:18:09.009490 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:09.009408 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" event={"ID":"a1a5a8ea-d576-42a5-a612-fd894d234b40","Type":"ContainerDied","Data":"db80ae435ced91d45c81702077ffa60d9aec767d0a884f4474add711d50af52e"} Sep 08 16:18:09.009720 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:09.009703 2555 scope.go:117] "RemoveContainer" containerID="db80ae435ced91d45c81702077ffa60d9aec767d0a884f4474add711d50af52e" Sep 08 16:18:10.000889 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:10.000844 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-765jc_b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008/serve-healthcheck-canary/0.log" Sep 08 16:18:10.015895 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:10.015859 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-694cc5f4dd-czffm" event={"ID":"4f36e387-c3ae-47c5-81cd-ace60aca2f38","Type":"ContainerStarted","Data":"ce0a05f130cb2999fc78293897b05190ecf60207ab5034c1de1c79e73b9f019e"} Sep 08 16:18:10.017690 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:10.017660 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-65p48" event={"ID":"1e9c00eb-2239-4e76-9155-8bb822586051","Type":"ContainerStarted","Data":"a22771b4edf7cde2ca70531023773158285276dfd5a829f065455424a3c75312"} Sep 08 16:18:10.019822 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:10.019799 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srwhq" event={"ID":"a1a5a8ea-d576-42a5-a612-fd894d234b40","Type":"ContainerStarted","Data":"f04d49c68c4c5696ac231f470b94989dd408a35db23e5235faa1d89cfcd97b8b"} Sep 08 16:18:39.235708 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.235623 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:18:39.244922 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.244885 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.249130 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.249097 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 08 16:18:39.249601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.249575 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 08 16:18:39.249809 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.249786 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 08 16:18:39.249891 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.249829 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 08 16:18:39.249976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.249600 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 08 16:18:39.250059 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.250041 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-prntc\"" Sep 08 16:18:39.250116 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.249596 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 08 16:18:39.250369 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.250323 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 08 16:18:39.250464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.250371 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 08 16:18:39.256981 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.256952 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:18:39.257930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.257912 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.257941 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.257960 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.257987 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258121 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258050 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-config-out\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258121 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258079 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-web-config\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258121 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258096 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258135 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258163 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258181 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258198 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2d2\" (UniqueName: \"kubernetes.io/projected/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-kube-api-access-hz2d2\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258223 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.258303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.258260 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.265640 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.265603 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 08 16:18:39.359129 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359129 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359135 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359165 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359188 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2d2\" (UniqueName: \"kubernetes.io/projected/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-kube-api-access-hz2d2\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359213 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359270 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359301 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359327 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359353 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359393 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359426 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-config-out\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359882 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359460 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-web-config\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359882 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359484 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.359989 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.359879 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.360191 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.360162 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.360395 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.360305 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.362444 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.362416 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-config-out\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.362562 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.362483 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.362624 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.362605 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.362695 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.362616 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.362954 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.362936 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.362954 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.362944 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-web-config\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.363083 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.362973 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.363232 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.363209 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.364453 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.364432 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.369686 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.369656 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2d2\" (UniqueName: \"kubernetes.io/projected/662cc3a7-0499-4eda-82f3-daf4b1cda0f2-kube-api-access-hz2d2\") pod \"alertmanager-main-0\" (UID: \"662cc3a7-0499-4eda-82f3-daf4b1cda0f2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.555501 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.555414 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:18:39.837512 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:18:39.837478 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662cc3a7_0499_4eda_82f3_daf4b1cda0f2.slice/crio-a7e778da0d90bab3089701cc6e149442a8d9e37f9caea079253e33311a1f43ef WatchSource:0}: Error finding container a7e778da0d90bab3089701cc6e149442a8d9e37f9caea079253e33311a1f43ef: Status 404 returned error can't find the container with id a7e778da0d90bab3089701cc6e149442a8d9e37f9caea079253e33311a1f43ef Sep 08 16:18:39.839607 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.839588 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:18:39.842929 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:39.842905 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:18:40.134890 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:40.134847 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerStarted","Data":"a7e778da0d90bab3089701cc6e149442a8d9e37f9caea079253e33311a1f43ef"} Sep 08 16:18:41.065024 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.064984 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7488989b8-g42mz"] Sep 08 16:18:41.068498 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.068470 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.070561 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070529 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-metrics-client-ca\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.070719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070590 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t27j\" (UniqueName: \"kubernetes.io/projected/b25f561f-2371-4a04-b73b-4f38cc88fa37-kube-api-access-8t27j\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.070719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070654 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-telemeter-client-tls\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.070719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070713 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-serving-certs-ca-bundle\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.070888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070774 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.070888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070824 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-secret-telemeter-client\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.070888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070867 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.071012 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.070890 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-federate-client-tls\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.075874 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.075842 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 08 16:18:41.075874 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.075873 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7488989b8-g42mz"] Sep 08 16:18:41.079615 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.079589 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 08 16:18:41.079909 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.079887 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 08 16:18:41.080042 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.079911 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 08 16:18:41.080042 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.079916 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 08 16:18:41.080042 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.079945 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s8zhf\"" Sep 08 16:18:41.094425 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.094391 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 08 16:18:41.140775 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.140742 2555 generic.go:356] "Generic (PLEG): container finished" podID="662cc3a7-0499-4eda-82f3-daf4b1cda0f2" containerID="afcfe2e032a0922f5ae2c2fde097410880e35e4080ce712947f21920b372abea" exitCode=0 Sep 08 16:18:41.140948 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.140832 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerDied","Data":"afcfe2e032a0922f5ae2c2fde097410880e35e4080ce712947f21920b372abea"} Sep 08 16:18:41.172525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t27j\" (UniqueName: \"kubernetes.io/projected/b25f561f-2371-4a04-b73b-4f38cc88fa37-kube-api-access-8t27j\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.172525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172542 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-telemeter-client-tls\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.172856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-serving-certs-ca-bundle\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.172856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172660 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.172856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-secret-telemeter-client\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.172856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.172856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-federate-client-tls\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.173144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.172859 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-metrics-client-ca\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.174051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.173738 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-serving-certs-ca-bundle\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.174051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.173842 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-metrics-client-ca\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.174623 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.174595 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b25f561f-2371-4a04-b73b-4f38cc88fa37-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.175647 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.175625 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-telemeter-client-tls\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.175767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.175745 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.175888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.175867 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-federate-client-tls\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.176515 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.176490 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b25f561f-2371-4a04-b73b-4f38cc88fa37-secret-telemeter-client\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.186566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.186535 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t27j\" (UniqueName: \"kubernetes.io/projected/b25f561f-2371-4a04-b73b-4f38cc88fa37-kube-api-access-8t27j\") pod \"telemeter-client-7488989b8-g42mz\" (UID: \"b25f561f-2371-4a04-b73b-4f38cc88fa37\") " pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.417341 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.417302 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" Sep 08 16:18:41.697722 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:18:41.697694 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25f561f_2371_4a04_b73b_4f38cc88fa37.slice/crio-3491cd0f1aa99e69497f519175d8f2ea342a24c6955d87bcfe2946c3b3a5775b WatchSource:0}: Error finding container 3491cd0f1aa99e69497f519175d8f2ea342a24c6955d87bcfe2946c3b3a5775b: Status 404 returned error can't find the container with id 3491cd0f1aa99e69497f519175d8f2ea342a24c6955d87bcfe2946c3b3a5775b Sep 08 16:18:41.703622 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:41.703594 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7488989b8-g42mz"] Sep 08 16:18:42.151439 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:42.151397 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" event={"ID":"b25f561f-2371-4a04-b73b-4f38cc88fa37","Type":"ContainerStarted","Data":"3491cd0f1aa99e69497f519175d8f2ea342a24c6955d87bcfe2946c3b3a5775b"} Sep 08 16:18:44.165726 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.165687 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerStarted","Data":"7a1a98535e1605e0ef6604ce5e567b24b6c2b2b0366a2dcb4e773e1e3977e5f5"} Sep 08 16:18:44.166106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.165769 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerStarted","Data":"5de5e85c5038e3bcdac62ac8f42a49d100fc4fdd482ec0edda3cb617ee2ed34c"} Sep 08 16:18:44.166106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.165796 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerStarted","Data":"ef60947ca882e806e1571b8d9f57e73866371df5126cf175436ad7b6dd424fee"} Sep 08 16:18:44.166106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.165819 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerStarted","Data":"ccc35f81b1c61011453edce500d862f53d3e291b36999c6fc8194f087067769a"} Sep 08 16:18:44.166106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.165842 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerStarted","Data":"0bcdeb190f6624c488b1250084efae18fdb700fe422b66d80b72006edb060ed0"} Sep 08 16:18:44.167632 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.167607 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" event={"ID":"b25f561f-2371-4a04-b73b-4f38cc88fa37","Type":"ContainerStarted","Data":"f723200edd24e642a9a05be02dbc7947474d077dc45009a73b053f023c0980e6"} Sep 08 16:18:44.167680 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.167662 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" event={"ID":"b25f561f-2371-4a04-b73b-4f38cc88fa37","Type":"ContainerStarted","Data":"d6d3f1d92ddbb855b16da19b6274582194790a6e45f693f5a7982bbeaadde204"} Sep 08 16:18:44.167714 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.167692 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" event={"ID":"b25f561f-2371-4a04-b73b-4f38cc88fa37","Type":"ContainerStarted","Data":"10eec04619c59efa3b1d9359453fc266c5cf20a21891e45e885d546cbc4a45c3"} Sep 08 16:18:44.206979 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:44.206919 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7488989b8-g42mz" podStartSLOduration=1.270278028 podStartE2EDuration="3.206903072s" totalImagesPullingTime="1.936625044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:18:44.205646682 +0000 UTC m=+357.800645460" watchObservedRunningTime="2026-09-08 16:18:44.206903072 +0000 UTC m=+357.801901830" Sep 08 16:18:45.175549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:45.175514 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"662cc3a7-0499-4eda-82f3-daf4b1cda0f2","Type":"ContainerStarted","Data":"30a63b16bb5b07f14a5806d2da821892a3a29e0b994b4929b53ea7b99721f658"} Sep 08 16:18:45.224113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:45.222255 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.184877358 podStartE2EDuration="6.222218029s" totalImagesPullingTime="4.017865358s" totalInitContainerRuntime="19.475313ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:18:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:18:45.219616494 +0000 UTC m=+358.814615264" watchObservedRunningTime="2026-09-08 16:18:45.222218029 +0000 UTC m=+358.817216791" Sep 08 16:18:54.851954 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:18:54.851913 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6598cc7fc5-g5j7f"] Sep 08 16:19:19.872641 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:19.872594 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6598cc7fc5-g5j7f" podUID="98c3200d-83e3-45e1-be2c-2395ff6d7b3f" containerName="console" containerID="cri-o://5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897" gracePeriod=15 Sep 08 16:19:20.127600 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.127531 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:19:20.207174 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207136 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-oauth-serving-cert\") pod \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " Sep 08 16:19:20.207174 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207181 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-oauth-config\") pod \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " Sep 08 16:19:20.207423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207216 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-service-ca\") pod \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " Sep 08 16:19:20.207423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207389 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-serving-cert\") pod \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " Sep 08 16:19:20.207519 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207457 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp4x2\" (UniqueName: \"kubernetes.io/projected/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-kube-api-access-vp4x2\") pod \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " Sep 08 16:19:20.207669 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207642 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-config\") pod \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " Sep 08 16:19:20.207788 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207642 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-service-ca" (OuterVolumeSpecName: "service-ca") pod "98c3200d-83e3-45e1-be2c-2395ff6d7b3f" (UID: "98c3200d-83e3-45e1-be2c-2395ff6d7b3f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:19:20.207788 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207699 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-trusted-ca-bundle\") pod \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\" (UID: \"98c3200d-83e3-45e1-be2c-2395ff6d7b3f\") " Sep 08 16:19:20.207788 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207651 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "98c3200d-83e3-45e1-be2c-2395ff6d7b3f" (UID: "98c3200d-83e3-45e1-be2c-2395ff6d7b3f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:19:20.207963 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207946 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-oauth-serving-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:19:20.208011 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.207970 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:19:20.208059 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.208041 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-config" (OuterVolumeSpecName: "console-config") pod "98c3200d-83e3-45e1-be2c-2395ff6d7b3f" (UID: "98c3200d-83e3-45e1-be2c-2395ff6d7b3f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:19:20.208182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.208154 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "98c3200d-83e3-45e1-be2c-2395ff6d7b3f" (UID: "98c3200d-83e3-45e1-be2c-2395ff6d7b3f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:19:20.209721 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.209694 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-kube-api-access-vp4x2" (OuterVolumeSpecName: "kube-api-access-vp4x2") pod "98c3200d-83e3-45e1-be2c-2395ff6d7b3f" (UID: "98c3200d-83e3-45e1-be2c-2395ff6d7b3f"). InnerVolumeSpecName "kube-api-access-vp4x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:19:20.209932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.209895 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "98c3200d-83e3-45e1-be2c-2395ff6d7b3f" (UID: "98c3200d-83e3-45e1-be2c-2395ff6d7b3f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:19:20.210028 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.209968 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "98c3200d-83e3-45e1-be2c-2395ff6d7b3f" (UID: "98c3200d-83e3-45e1-be2c-2395ff6d7b3f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:19:20.308740 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.308686 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-oauth-config\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:19:20.308740 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.308733 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-serving-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:19:20.308740 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.308745 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp4x2\" (UniqueName: \"kubernetes.io/projected/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-kube-api-access-vp4x2\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:19:20.308740 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.308755 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-console-config\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:19:20.309001 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.308763 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c3200d-83e3-45e1-be2c-2395ff6d7b3f-trusted-ca-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:19:20.320720 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.320689 2555 generic.go:356] "Generic (PLEG): container finished" podID="98c3200d-83e3-45e1-be2c-2395ff6d7b3f" containerID="5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897" exitCode=0 Sep 08 16:19:20.320845 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.320740 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6598cc7fc5-g5j7f" Sep 08 16:19:20.320845 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.320783 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6598cc7fc5-g5j7f" event={"ID":"98c3200d-83e3-45e1-be2c-2395ff6d7b3f","Type":"ContainerDied","Data":"5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897"} Sep 08 16:19:20.320951 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.320883 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6598cc7fc5-g5j7f" event={"ID":"98c3200d-83e3-45e1-be2c-2395ff6d7b3f","Type":"ContainerDied","Data":"644d8e487245492ea3af01393b6ea77fe2997ee547353c96b9dfa91a448bc906"} Sep 08 16:19:20.320951 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.320917 2555 scope.go:117] "RemoveContainer" containerID="5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897" Sep 08 16:19:20.330297 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.330278 2555 scope.go:117] "RemoveContainer" containerID="5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897" Sep 08 16:19:20.330603 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:19:20.330583 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897\": container with ID starting with 5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897 not found: ID does not exist" containerID="5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897" Sep 08 16:19:20.330661 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.330614 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897"} err="failed to get container status \"5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897\": rpc error: code = NotFound desc = could not find container \"5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897\": container with ID starting with 5fd2dd23c9897cd4905cff38ae3fb5d89cac16292b325cb4015ad454c7f01897 not found: ID does not exist" Sep 08 16:19:20.341924 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.341891 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6598cc7fc5-g5j7f"] Sep 08 16:19:20.346142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.346114 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6598cc7fc5-g5j7f"] Sep 08 16:19:20.935952 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:19:20.935917 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c3200d-83e3-45e1-be2c-2395ff6d7b3f" path="/var/lib/kubelet/pods/98c3200d-83e3-45e1-be2c-2395ff6d7b3f/volumes" Sep 08 16:20:20.404384 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.404291 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c76b748c5-gs8mw"] Sep 08 16:20:20.404965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.404768 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c3200d-83e3-45e1-be2c-2395ff6d7b3f" containerName="console" Sep 08 16:20:20.404965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.404787 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c3200d-83e3-45e1-be2c-2395ff6d7b3f" containerName="console" Sep 08 16:20:20.404965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.404913 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="98c3200d-83e3-45e1-be2c-2395ff6d7b3f" containerName="console" Sep 08 16:20:20.407766 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.407744 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.410361 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.410337 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 16:20:20.410530 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.410517 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 16:20:20.410530 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.410522 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 16:20:20.411388 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.411371 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 16:20:20.411858 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.411838 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 16:20:20.411975 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.411873 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 16:20:20.411975 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.411904 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 16:20:20.412113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.412015 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dz7wm\"" Sep 08 16:20:20.416043 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.416021 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 16:20:20.418502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.418475 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c76b748c5-gs8mw"] Sep 08 16:20:20.492362 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.492323 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-oauth-config\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.492362 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.492363 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-oauth-serving-cert\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.492596 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.492395 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-service-ca\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.492596 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.492447 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-serving-cert\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.492596 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.492476 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-trusted-ca-bundle\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.492596 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.492525 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdck\" (UniqueName: \"kubernetes.io/projected/22b4d105-b934-4f48-ae54-7505be206cd7-kube-api-access-7qdck\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.492741 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.492595 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-console-config\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.593183 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593150 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdck\" (UniqueName: \"kubernetes.io/projected/22b4d105-b934-4f48-ae54-7505be206cd7-kube-api-access-7qdck\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.593414 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593202 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-console-config\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.593414 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593326 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-oauth-config\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.593414 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593359 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-oauth-serving-cert\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.593594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593469 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-service-ca\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.593594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593502 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-serving-cert\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.593594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593524 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-trusted-ca-bundle\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.594003 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.593983 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-console-config\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.594099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.594084 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-oauth-serving-cert\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.594157 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.594107 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-service-ca\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.594423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.594390 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-trusted-ca-bundle\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.595912 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.595888 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-oauth-config\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.596000 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.595970 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-serving-cert\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.600482 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.600454 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdck\" (UniqueName: \"kubernetes.io/projected/22b4d105-b934-4f48-ae54-7505be206cd7-kube-api-access-7qdck\") pod \"console-5c76b748c5-gs8mw\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.717933 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.717834 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:20.993438 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:20.993209 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c76b748c5-gs8mw"] Sep 08 16:20:20.996158 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:20:20.996126 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b4d105_b934_4f48_ae54_7505be206cd7.slice/crio-0038f17f021b8f0708b82bebb0bb2a0761618bbb5e4336964bf859da47245632 WatchSource:0}: Error finding container 0038f17f021b8f0708b82bebb0bb2a0761618bbb5e4336964bf859da47245632: Status 404 returned error can't find the container with id 0038f17f021b8f0708b82bebb0bb2a0761618bbb5e4336964bf859da47245632 Sep 08 16:20:21.560067 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:21.560027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c76b748c5-gs8mw" event={"ID":"22b4d105-b934-4f48-ae54-7505be206cd7","Type":"ContainerStarted","Data":"087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73"} Sep 08 16:20:21.560563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:21.560083 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c76b748c5-gs8mw" event={"ID":"22b4d105-b934-4f48-ae54-7505be206cd7","Type":"ContainerStarted","Data":"0038f17f021b8f0708b82bebb0bb2a0761618bbb5e4336964bf859da47245632"} Sep 08 16:20:30.718352 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:30.718291 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:30.718768 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:30.718394 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:30.723193 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:30.723166 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:20:30.739748 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:30.739693 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c76b748c5-gs8mw" podStartSLOduration=10.739676812999999 podStartE2EDuration="10.739676813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:20:21.575463169 +0000 UTC m=+455.170461954" watchObservedRunningTime="2026-09-08 16:20:30.739676813 +0000 UTC m=+464.334675573" Sep 08 16:20:31.604074 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:20:31.604043 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:21:04.109211 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.109174 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t"] Sep 08 16:21:04.112739 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.112720 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.115192 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.115168 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:21:04.116037 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.116021 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9vsgj\"" Sep 08 16:21:04.116095 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.116068 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:21:04.121854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.121824 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t"] Sep 08 16:21:04.233746 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.233711 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.233934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.233757 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpjq\" (UniqueName: \"kubernetes.io/projected/4892e6d2-197f-4ec6-bc3f-ded6e7150865-kube-api-access-bcpjq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.233934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.233797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.334558 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.334510 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.334722 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.334597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.334722 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.334640 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpjq\" (UniqueName: \"kubernetes.io/projected/4892e6d2-197f-4ec6-bc3f-ded6e7150865-kube-api-access-bcpjq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.334984 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.334961 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.335049 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.334982 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.342824 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.342800 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpjq\" (UniqueName: \"kubernetes.io/projected/4892e6d2-197f-4ec6-bc3f-ded6e7150865-kube-api-access-bcpjq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.422588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.422487 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:04.690048 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.689966 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t"] Sep 08 16:21:04.693171 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:21:04.693143 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4892e6d2_197f_4ec6_bc3f_ded6e7150865.slice/crio-302e6e99b9529dcfa7f54bfa3887a2b31d62dd98a46a0fe5a48dff8653669780 WatchSource:0}: Error finding container 302e6e99b9529dcfa7f54bfa3887a2b31d62dd98a46a0fe5a48dff8653669780: Status 404 returned error can't find the container with id 302e6e99b9529dcfa7f54bfa3887a2b31d62dd98a46a0fe5a48dff8653669780 Sep 08 16:21:04.729132 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:04.729100 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" event={"ID":"4892e6d2-197f-4ec6-bc3f-ded6e7150865","Type":"ContainerStarted","Data":"302e6e99b9529dcfa7f54bfa3887a2b31d62dd98a46a0fe5a48dff8653669780"} Sep 08 16:21:12.766462 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:12.766374 2555 generic.go:356] "Generic (PLEG): container finished" podID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerID="320cd76d8986ecbef6171747d2a6ca43e8b4aafe83b5e0b1dcd2ccfe07b4852b" exitCode=0 Sep 08 16:21:12.766462 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:12.766435 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" event={"ID":"4892e6d2-197f-4ec6-bc3f-ded6e7150865","Type":"ContainerDied","Data":"320cd76d8986ecbef6171747d2a6ca43e8b4aafe83b5e0b1dcd2ccfe07b4852b"} Sep 08 16:21:16.786633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:16.786595 2555 generic.go:356] "Generic (PLEG): container finished" podID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerID="e7658ecf272b11d36478489a7ed66db86ab1c55f394aedef51244bacd6837502" exitCode=0 Sep 08 16:21:16.787021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:16.786678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" event={"ID":"4892e6d2-197f-4ec6-bc3f-ded6e7150865","Type":"ContainerDied","Data":"e7658ecf272b11d36478489a7ed66db86ab1c55f394aedef51244bacd6837502"} Sep 08 16:21:29.841276 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:29.841210 2555 generic.go:356] "Generic (PLEG): container finished" podID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerID="81fe3427df7420de8d353a4c8eee9a740e563052b43fff41aaf12a9af1596cec" exitCode=0 Sep 08 16:21:29.841686 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:29.841297 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" event={"ID":"4892e6d2-197f-4ec6-bc3f-ded6e7150865","Type":"ContainerDied","Data":"81fe3427df7420de8d353a4c8eee9a740e563052b43fff41aaf12a9af1596cec"} Sep 08 16:21:30.968698 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:30.968667 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:31.066748 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.066711 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcpjq\" (UniqueName: \"kubernetes.io/projected/4892e6d2-197f-4ec6-bc3f-ded6e7150865-kube-api-access-bcpjq\") pod \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " Sep 08 16:21:31.066933 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.066774 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-util\") pod \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " Sep 08 16:21:31.066933 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.066839 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-bundle\") pod \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\" (UID: \"4892e6d2-197f-4ec6-bc3f-ded6e7150865\") " Sep 08 16:21:31.067513 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.067482 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-bundle" (OuterVolumeSpecName: "bundle") pod "4892e6d2-197f-4ec6-bc3f-ded6e7150865" (UID: "4892e6d2-197f-4ec6-bc3f-ded6e7150865"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:21:31.068972 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.068949 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4892e6d2-197f-4ec6-bc3f-ded6e7150865-kube-api-access-bcpjq" (OuterVolumeSpecName: "kube-api-access-bcpjq") pod "4892e6d2-197f-4ec6-bc3f-ded6e7150865" (UID: "4892e6d2-197f-4ec6-bc3f-ded6e7150865"). InnerVolumeSpecName "kube-api-access-bcpjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:21:31.071239 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.071216 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-util" (OuterVolumeSpecName: "util") pod "4892e6d2-197f-4ec6-bc3f-ded6e7150865" (UID: "4892e6d2-197f-4ec6-bc3f-ded6e7150865"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:21:31.167454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.167374 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:21:31.167454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.167405 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcpjq\" (UniqueName: \"kubernetes.io/projected/4892e6d2-197f-4ec6-bc3f-ded6e7150865-kube-api-access-bcpjq\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:21:31.167454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.167414 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4892e6d2-197f-4ec6-bc3f-ded6e7150865-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:21:31.851134 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.851105 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" Sep 08 16:21:31.851134 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.851129 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b7m7t" event={"ID":"4892e6d2-197f-4ec6-bc3f-ded6e7150865","Type":"ContainerDied","Data":"302e6e99b9529dcfa7f54bfa3887a2b31d62dd98a46a0fe5a48dff8653669780"} Sep 08 16:21:31.851366 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:31.851173 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="302e6e99b9529dcfa7f54bfa3887a2b31d62dd98a46a0fe5a48dff8653669780" Sep 08 16:21:37.090708 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.090666 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l"] Sep 08 16:21:37.091119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.090991 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerName="util" Sep 08 16:21:37.091119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.091003 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerName="util" Sep 08 16:21:37.091119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.091011 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerName="pull" Sep 08 16:21:37.091119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.091016 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerName="pull" Sep 08 16:21:37.091119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.091026 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerName="extract" Sep 08 16:21:37.091119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.091032 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerName="extract" Sep 08 16:21:37.091119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.091084 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4892e6d2-197f-4ec6-bc3f-ded6e7150865" containerName="extract" Sep 08 16:21:37.093088 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.093071 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.095552 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.095528 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:21:37.095700 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.095530 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dbtvb\"" Sep 08 16:21:37.095892 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.095872 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 08 16:21:37.104962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.104935 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l"] Sep 08 16:21:37.112793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.112758 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbbz2\" (UniqueName: \"kubernetes.io/projected/d3e95e8b-1097-4fa6-a7ba-dcc7df23820d-kube-api-access-qbbz2\") pod \"cert-manager-operator-controller-manager-55c444bf86-vfc8l\" (UID: \"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.112930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.112839 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3e95e8b-1097-4fa6-a7ba-dcc7df23820d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vfc8l\" (UID: \"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.213987 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.213941 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbbz2\" (UniqueName: \"kubernetes.io/projected/d3e95e8b-1097-4fa6-a7ba-dcc7df23820d-kube-api-access-qbbz2\") pod \"cert-manager-operator-controller-manager-55c444bf86-vfc8l\" (UID: \"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.214202 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.213999 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3e95e8b-1097-4fa6-a7ba-dcc7df23820d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vfc8l\" (UID: \"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.214411 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.214392 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3e95e8b-1097-4fa6-a7ba-dcc7df23820d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vfc8l\" (UID: \"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.222844 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.222823 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbbz2\" (UniqueName: \"kubernetes.io/projected/d3e95e8b-1097-4fa6-a7ba-dcc7df23820d-kube-api-access-qbbz2\") pod \"cert-manager-operator-controller-manager-55c444bf86-vfc8l\" (UID: \"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.402358 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.402307 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" Sep 08 16:21:37.691106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.690970 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l"] Sep 08 16:21:37.694372 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:21:37.694338 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e95e8b_1097_4fa6_a7ba_dcc7df23820d.slice/crio-5b9eb0f715158116531ffc342eae1581de65691ad7232f729db004831531065d WatchSource:0}: Error finding container 5b9eb0f715158116531ffc342eae1581de65691ad7232f729db004831531065d: Status 404 returned error can't find the container with id 5b9eb0f715158116531ffc342eae1581de65691ad7232f729db004831531065d Sep 08 16:21:37.876466 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:37.876430 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" event={"ID":"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d","Type":"ContainerStarted","Data":"5b9eb0f715158116531ffc342eae1581de65691ad7232f729db004831531065d"} Sep 08 16:21:39.889045 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:39.888999 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" event={"ID":"d3e95e8b-1097-4fa6-a7ba-dcc7df23820d","Type":"ContainerStarted","Data":"1fbee6926d0f3f25c2f899b72e4ae1cfad1a832ad228d26b64b53c21d311048e"} Sep 08 16:21:39.910260 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:39.910191 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vfc8l" podStartSLOduration=1.481323917 podStartE2EDuration="2.910173458s" totalImagesPullingTime="1.428849541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:21:39.908547708 +0000 UTC m=+533.503546468" watchObservedRunningTime="2026-09-08 16:21:39.910173458 +0000 UTC m=+533.505172216" Sep 08 16:21:40.848545 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.848505 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx"] Sep 08 16:21:40.850924 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.850905 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:40.853441 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.853414 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:21:40.854328 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.854309 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:21:40.854440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.854314 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9vsgj\"" Sep 08 16:21:40.862331 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.862303 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx"] Sep 08 16:21:40.949401 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.949361 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpfh\" (UniqueName: \"kubernetes.io/projected/9b138d38-13d5-4edb-8e74-b7e106346671-kube-api-access-wwpfh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:40.949772 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.949497 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:40.949772 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:40.949542 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.050618 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.050579 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.050819 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.050625 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.050819 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.050688 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpfh\" (UniqueName: \"kubernetes.io/projected/9b138d38-13d5-4edb-8e74-b7e106346671-kube-api-access-wwpfh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.051102 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.051079 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.051155 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.051115 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.069866 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.069827 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpfh\" (UniqueName: \"kubernetes.io/projected/9b138d38-13d5-4edb-8e74-b7e106346671-kube-api-access-wwpfh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.160163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.160084 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:41.438299 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.438204 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx"] Sep 08 16:21:41.441605 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:21:41.441580 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b138d38_13d5_4edb_8e74_b7e106346671.slice/crio-6c81500a5817dc5688cbc6ec31ac45c2c78ec01e81793808b3b98d104f1c5b11 WatchSource:0}: Error finding container 6c81500a5817dc5688cbc6ec31ac45c2c78ec01e81793808b3b98d104f1c5b11: Status 404 returned error can't find the container with id 6c81500a5817dc5688cbc6ec31ac45c2c78ec01e81793808b3b98d104f1c5b11 Sep 08 16:21:41.905412 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.905377 2555 generic.go:356] "Generic (PLEG): container finished" podID="9b138d38-13d5-4edb-8e74-b7e106346671" containerID="3ab12f54b725a2494ea236637fa69323208fe518b14cb6f40804f71ba6f1ec58" exitCode=0 Sep 08 16:21:41.905598 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.905465 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" event={"ID":"9b138d38-13d5-4edb-8e74-b7e106346671","Type":"ContainerDied","Data":"3ab12f54b725a2494ea236637fa69323208fe518b14cb6f40804f71ba6f1ec58"} Sep 08 16:21:41.905598 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:41.905517 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" event={"ID":"9b138d38-13d5-4edb-8e74-b7e106346671","Type":"ContainerStarted","Data":"6c81500a5817dc5688cbc6ec31ac45c2c78ec01e81793808b3b98d104f1c5b11"} Sep 08 16:21:43.363956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.363914 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl"] Sep 08 16:21:43.367107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.367089 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.369657 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.369633 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 16:21:43.369792 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.369635 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vvtt9\"" Sep 08 16:21:43.370550 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.370534 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 16:21:43.376750 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.376724 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl"] Sep 08 16:21:43.471743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.471698 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1be1975-404f-4f1c-91b2-d6a91d040899-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xxwvl\" (UID: \"a1be1975-404f-4f1c-91b2-d6a91d040899\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.471920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.471762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqc6\" (UniqueName: \"kubernetes.io/projected/a1be1975-404f-4f1c-91b2-d6a91d040899-kube-api-access-hmqc6\") pod \"cert-manager-cainjector-559cdd8c87-xxwvl\" (UID: \"a1be1975-404f-4f1c-91b2-d6a91d040899\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.572580 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.572536 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1be1975-404f-4f1c-91b2-d6a91d040899-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xxwvl\" (UID: \"a1be1975-404f-4f1c-91b2-d6a91d040899\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.572744 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.572590 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqc6\" (UniqueName: \"kubernetes.io/projected/a1be1975-404f-4f1c-91b2-d6a91d040899-kube-api-access-hmqc6\") pod \"cert-manager-cainjector-559cdd8c87-xxwvl\" (UID: \"a1be1975-404f-4f1c-91b2-d6a91d040899\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.580785 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.580752 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1be1975-404f-4f1c-91b2-d6a91d040899-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xxwvl\" (UID: \"a1be1975-404f-4f1c-91b2-d6a91d040899\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.581044 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.581021 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqc6\" (UniqueName: \"kubernetes.io/projected/a1be1975-404f-4f1c-91b2-d6a91d040899-kube-api-access-hmqc6\") pod \"cert-manager-cainjector-559cdd8c87-xxwvl\" (UID: \"a1be1975-404f-4f1c-91b2-d6a91d040899\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.677161 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.677068 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" Sep 08 16:21:43.948145 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:43.948116 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl"] Sep 08 16:21:43.951053 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:21:43.951013 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1be1975_404f_4f1c_91b2_d6a91d040899.slice/crio-106aff15f9f869bb9512231336f7c2bb7ed0293f96e2c204d3993c934d79574e WatchSource:0}: Error finding container 106aff15f9f869bb9512231336f7c2bb7ed0293f96e2c204d3993c934d79574e: Status 404 returned error can't find the container with id 106aff15f9f869bb9512231336f7c2bb7ed0293f96e2c204d3993c934d79574e Sep 08 16:21:44.920818 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:44.920737 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" event={"ID":"a1be1975-404f-4f1c-91b2-d6a91d040899","Type":"ContainerStarted","Data":"106aff15f9f869bb9512231336f7c2bb7ed0293f96e2c204d3993c934d79574e"} Sep 08 16:21:45.928319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:45.928273 2555 generic.go:356] "Generic (PLEG): container finished" podID="9b138d38-13d5-4edb-8e74-b7e106346671" containerID="ac0fbc485a49269efed1a9eeaa03bee6fe57e87a9cd2167e8ac4e6b59c28bc86" exitCode=0 Sep 08 16:21:45.928813 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:45.928335 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" event={"ID":"9b138d38-13d5-4edb-8e74-b7e106346671","Type":"ContainerDied","Data":"ac0fbc485a49269efed1a9eeaa03bee6fe57e87a9cd2167e8ac4e6b59c28bc86"} Sep 08 16:21:46.936957 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:46.936919 2555 generic.go:356] "Generic (PLEG): container finished" podID="9b138d38-13d5-4edb-8e74-b7e106346671" containerID="1c4273d12aace8539342dc9599de6f1451a8fd49a14b6f8dcf8f2b18d7288bbd" exitCode=0 Sep 08 16:21:46.937741 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:46.937718 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" event={"ID":"9b138d38-13d5-4edb-8e74-b7e106346671","Type":"ContainerDied","Data":"1c4273d12aace8539342dc9599de6f1451a8fd49a14b6f8dcf8f2b18d7288bbd"} Sep 08 16:21:47.943947 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:47.943907 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" event={"ID":"a1be1975-404f-4f1c-91b2-d6a91d040899","Type":"ContainerStarted","Data":"a8e05a542de712aecc49d3bc5dbd387ec99ca367cb8a79640bb1a00aa18bbf04"} Sep 08 16:21:47.960994 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:47.960921 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xxwvl" podStartSLOduration=1.324743727 podStartE2EDuration="4.960899506s" totalImagesPullingTime="3.636155779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:21:47.958656926 +0000 UTC m=+541.553655710" watchObservedRunningTime="2026-09-08 16:21:47.960899506 +0000 UTC m=+541.555898267" Sep 08 16:21:48.086751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.086723 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:48.209775 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.209679 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-bundle\") pod \"9b138d38-13d5-4edb-8e74-b7e106346671\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " Sep 08 16:21:48.209944 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.209812 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwpfh\" (UniqueName: \"kubernetes.io/projected/9b138d38-13d5-4edb-8e74-b7e106346671-kube-api-access-wwpfh\") pod \"9b138d38-13d5-4edb-8e74-b7e106346671\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " Sep 08 16:21:48.209944 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.209851 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-util\") pod \"9b138d38-13d5-4edb-8e74-b7e106346671\" (UID: \"9b138d38-13d5-4edb-8e74-b7e106346671\") " Sep 08 16:21:48.210194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.210169 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-bundle" (OuterVolumeSpecName: "bundle") pod "9b138d38-13d5-4edb-8e74-b7e106346671" (UID: "9b138d38-13d5-4edb-8e74-b7e106346671"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:21:48.212124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.212089 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b138d38-13d5-4edb-8e74-b7e106346671-kube-api-access-wwpfh" (OuterVolumeSpecName: "kube-api-access-wwpfh") pod "9b138d38-13d5-4edb-8e74-b7e106346671" (UID: "9b138d38-13d5-4edb-8e74-b7e106346671"). InnerVolumeSpecName "kube-api-access-wwpfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:21:48.214583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.214555 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-util" (OuterVolumeSpecName: "util") pod "9b138d38-13d5-4edb-8e74-b7e106346671" (UID: "9b138d38-13d5-4edb-8e74-b7e106346671"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:21:48.310643 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.310587 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwpfh\" (UniqueName: \"kubernetes.io/projected/9b138d38-13d5-4edb-8e74-b7e106346671-kube-api-access-wwpfh\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:21:48.310643 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.310635 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:21:48.310643 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.310647 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b138d38-13d5-4edb-8e74-b7e106346671-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:21:48.950182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.950153 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" Sep 08 16:21:48.950627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.950200 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhlcqx" event={"ID":"9b138d38-13d5-4edb-8e74-b7e106346671","Type":"ContainerDied","Data":"6c81500a5817dc5688cbc6ec31ac45c2c78ec01e81793808b3b98d104f1c5b11"} Sep 08 16:21:48.950627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:48.950239 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c81500a5817dc5688cbc6ec31ac45c2c78ec01e81793808b3b98d104f1c5b11" Sep 08 16:21:54.732671 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.732637 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd"] Sep 08 16:21:54.733021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.732984 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b138d38-13d5-4edb-8e74-b7e106346671" containerName="pull" Sep 08 16:21:54.733021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.732996 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b138d38-13d5-4edb-8e74-b7e106346671" containerName="pull" Sep 08 16:21:54.733021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.733015 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b138d38-13d5-4edb-8e74-b7e106346671" containerName="extract" Sep 08 16:21:54.733021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.733022 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b138d38-13d5-4edb-8e74-b7e106346671" containerName="extract" Sep 08 16:21:54.733176 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.733036 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b138d38-13d5-4edb-8e74-b7e106346671" containerName="util" Sep 08 16:21:54.733176 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.733041 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b138d38-13d5-4edb-8e74-b7e106346671" containerName="util" Sep 08 16:21:54.733176 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.733088 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b138d38-13d5-4edb-8e74-b7e106346671" containerName="extract" Sep 08 16:21:54.737268 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.737236 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:54.739667 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.739646 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:21:54.740626 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.740605 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-s72gm\"" Sep 08 16:21:54.740712 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.740615 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 16:21:54.744925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.744895 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd"] Sep 08 16:21:54.867227 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.867185 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8928908-1aa8-4694-adf0-467fd7388683-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kl8nd\" (UID: \"b8928908-1aa8-4694-adf0-467fd7388683\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:54.867417 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.867266 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9z7\" (UniqueName: \"kubernetes.io/projected/b8928908-1aa8-4694-adf0-467fd7388683-kube-api-access-ts9z7\") pod \"openshift-lws-operator-fd8ccff4c-kl8nd\" (UID: \"b8928908-1aa8-4694-adf0-467fd7388683\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:54.967971 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.967941 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8928908-1aa8-4694-adf0-467fd7388683-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kl8nd\" (UID: \"b8928908-1aa8-4694-adf0-467fd7388683\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:54.968140 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.967993 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9z7\" (UniqueName: \"kubernetes.io/projected/b8928908-1aa8-4694-adf0-467fd7388683-kube-api-access-ts9z7\") pod \"openshift-lws-operator-fd8ccff4c-kl8nd\" (UID: \"b8928908-1aa8-4694-adf0-467fd7388683\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:54.968318 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.968299 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8928908-1aa8-4694-adf0-467fd7388683-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kl8nd\" (UID: \"b8928908-1aa8-4694-adf0-467fd7388683\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:54.980690 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:54.980655 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9z7\" (UniqueName: \"kubernetes.io/projected/b8928908-1aa8-4694-adf0-467fd7388683-kube-api-access-ts9z7\") pod \"openshift-lws-operator-fd8ccff4c-kl8nd\" (UID: \"b8928908-1aa8-4694-adf0-467fd7388683\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:55.047312 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:55.047234 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" Sep 08 16:21:55.317920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:55.317895 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd"] Sep 08 16:21:55.320746 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:21:55.320719 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8928908_1aa8_4694_adf0_467fd7388683.slice/crio-d1fd7a792c3987cb13ddf5440558ea35d286874be35934966ae57ab376c737c5 WatchSource:0}: Error finding container d1fd7a792c3987cb13ddf5440558ea35d286874be35934966ae57ab376c737c5: Status 404 returned error can't find the container with id d1fd7a792c3987cb13ddf5440558ea35d286874be35934966ae57ab376c737c5 Sep 08 16:21:55.985003 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:55.984967 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" event={"ID":"b8928908-1aa8-4694-adf0-467fd7388683","Type":"ContainerStarted","Data":"d1fd7a792c3987cb13ddf5440558ea35d286874be35934966ae57ab376c737c5"} Sep 08 16:21:57.995285 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:57.995235 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" event={"ID":"b8928908-1aa8-4694-adf0-467fd7388683","Type":"ContainerStarted","Data":"d5e1d8de486851e562e1be929cc3cd450e54694a8a5a2639f1470c19231d5cc1"} Sep 08 16:21:58.013348 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:21:58.013293 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kl8nd" podStartSLOduration=1.820324794 podStartE2EDuration="4.013275269s" totalImagesPullingTime="2.192950475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:21:58.01121102 +0000 UTC m=+551.606209792" watchObservedRunningTime="2026-09-08 16:21:58.013275269 +0000 UTC m=+551.608274029" Sep 08 16:22:00.707076 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.707040 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg"] Sep 08 16:22:00.728349 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.728315 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg"] Sep 08 16:22:00.728508 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.728431 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.730834 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.730810 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:22:00.731000 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.730926 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:22:00.731773 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.731752 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9vsgj\"" Sep 08 16:22:00.821562 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.821524 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7fj\" (UniqueName: \"kubernetes.io/projected/f0f40595-e64a-48f9-b09b-abc901fb6545-kube-api-access-tg7fj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.821562 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.821567 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.821781 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.821631 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.922790 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.922747 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7fj\" (UniqueName: \"kubernetes.io/projected/f0f40595-e64a-48f9-b09b-abc901fb6545-kube-api-access-tg7fj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.922976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.922797 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.922976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.922837 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.923334 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.923311 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.923386 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.923322 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:00.930681 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:00.930659 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7fj\" (UniqueName: \"kubernetes.io/projected/f0f40595-e64a-48f9-b09b-abc901fb6545-kube-api-access-tg7fj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:01.037708 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:01.037620 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:01.320150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:01.320125 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg"] Sep 08 16:22:01.322379 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:01.322346 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f40595_e64a_48f9_b09b_abc901fb6545.slice/crio-0d550a7df38848439bb61c1ea95f757921eb2759735f5a136f53443cda5ce5e8 WatchSource:0}: Error finding container 0d550a7df38848439bb61c1ea95f757921eb2759735f5a136f53443cda5ce5e8: Status 404 returned error can't find the container with id 0d550a7df38848439bb61c1ea95f757921eb2759735f5a136f53443cda5ce5e8 Sep 08 16:22:02.016237 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:02.016202 2555 generic.go:356] "Generic (PLEG): container finished" podID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerID="1b68d329ba1463b3c14849b03c46c763d4c87efd5cec3c8c8846bfbc89ff8d3b" exitCode=0 Sep 08 16:22:02.016624 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:02.016280 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" event={"ID":"f0f40595-e64a-48f9-b09b-abc901fb6545","Type":"ContainerDied","Data":"1b68d329ba1463b3c14849b03c46c763d4c87efd5cec3c8c8846bfbc89ff8d3b"} Sep 08 16:22:02.016624 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:02.016334 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" event={"ID":"f0f40595-e64a-48f9-b09b-abc901fb6545","Type":"ContainerStarted","Data":"0d550a7df38848439bb61c1ea95f757921eb2759735f5a136f53443cda5ce5e8"} Sep 08 16:22:03.022499 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:03.022464 2555 generic.go:356] "Generic (PLEG): container finished" podID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerID="c02270e923d1b099569390f84f7497c2ebee9a97d024f46074c181243c646ea7" exitCode=0 Sep 08 16:22:03.022942 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:03.022542 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" event={"ID":"f0f40595-e64a-48f9-b09b-abc901fb6545","Type":"ContainerDied","Data":"c02270e923d1b099569390f84f7497c2ebee9a97d024f46074c181243c646ea7"} Sep 08 16:22:04.029144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:04.029109 2555 generic.go:356] "Generic (PLEG): container finished" podID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerID="75960dd2ba401fde7f441a88036103f1b03c102d65f8631e13b98ee86d2f8082" exitCode=0 Sep 08 16:22:04.029561 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:04.029191 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" event={"ID":"f0f40595-e64a-48f9-b09b-abc901fb6545","Type":"ContainerDied","Data":"75960dd2ba401fde7f441a88036103f1b03c102d65f8631e13b98ee86d2f8082"} Sep 08 16:22:05.162496 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.162467 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:05.260532 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.260480 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-util\") pod \"f0f40595-e64a-48f9-b09b-abc901fb6545\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " Sep 08 16:22:05.260532 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.260548 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg7fj\" (UniqueName: \"kubernetes.io/projected/f0f40595-e64a-48f9-b09b-abc901fb6545-kube-api-access-tg7fj\") pod \"f0f40595-e64a-48f9-b09b-abc901fb6545\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " Sep 08 16:22:05.260770 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.260591 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-bundle\") pod \"f0f40595-e64a-48f9-b09b-abc901fb6545\" (UID: \"f0f40595-e64a-48f9-b09b-abc901fb6545\") " Sep 08 16:22:05.261288 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.261234 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-bundle" (OuterVolumeSpecName: "bundle") pod "f0f40595-e64a-48f9-b09b-abc901fb6545" (UID: "f0f40595-e64a-48f9-b09b-abc901fb6545"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:22:05.262732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.262692 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f40595-e64a-48f9-b09b-abc901fb6545-kube-api-access-tg7fj" (OuterVolumeSpecName: "kube-api-access-tg7fj") pod "f0f40595-e64a-48f9-b09b-abc901fb6545" (UID: "f0f40595-e64a-48f9-b09b-abc901fb6545"). InnerVolumeSpecName "kube-api-access-tg7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:22:05.265819 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.265778 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-util" (OuterVolumeSpecName: "util") pod "f0f40595-e64a-48f9-b09b-abc901fb6545" (UID: "f0f40595-e64a-48f9-b09b-abc901fb6545"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:22:05.362050 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.362004 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:22:05.362050 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.362048 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7fj\" (UniqueName: \"kubernetes.io/projected/f0f40595-e64a-48f9-b09b-abc901fb6545-kube-api-access-tg7fj\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:22:05.362050 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:05.362058 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f40595-e64a-48f9-b09b-abc901fb6545-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:22:06.040636 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:06.040606 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" Sep 08 16:22:06.040636 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:06.040625 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnsbpg" event={"ID":"f0f40595-e64a-48f9-b09b-abc901fb6545","Type":"ContainerDied","Data":"0d550a7df38848439bb61c1ea95f757921eb2759735f5a136f53443cda5ce5e8"} Sep 08 16:22:06.040909 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:06.040683 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d550a7df38848439bb61c1ea95f757921eb2759735f5a136f53443cda5ce5e8" Sep 08 16:22:17.026205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026169 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v"] Sep 08 16:22:17.026635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026490 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerName="util" Sep 08 16:22:17.026635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026502 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerName="util" Sep 08 16:22:17.026635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026529 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerName="pull" Sep 08 16:22:17.026635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026535 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerName="pull" Sep 08 16:22:17.026635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026543 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerName="extract" Sep 08 16:22:17.026635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026548 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerName="extract" Sep 08 16:22:17.026635 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.026598 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f40595-e64a-48f9-b09b-abc901fb6545" containerName="extract" Sep 08 16:22:17.028571 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.028554 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.031817 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.031797 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:22:17.031938 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.031894 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:22:17.032491 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.032473 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9vsgj\"" Sep 08 16:22:17.046382 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.046348 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v"] Sep 08 16:22:17.168135 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.168094 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.168317 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.168142 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crblf\" (UniqueName: \"kubernetes.io/projected/f168fd09-9fc9-42bd-9433-79f18e452fd2-kube-api-access-crblf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.168317 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.168226 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.269371 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.269331 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.269536 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.269384 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crblf\" (UniqueName: \"kubernetes.io/projected/f168fd09-9fc9-42bd-9433-79f18e452fd2-kube-api-access-crblf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.269536 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.269427 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.269715 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.269696 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.269845 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.269827 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.280521 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.280457 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crblf\" (UniqueName: \"kubernetes.io/projected/f168fd09-9fc9-42bd-9433-79f18e452fd2-kube-api-access-crblf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.337998 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.337959 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:17.616280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:17.616238 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v"] Sep 08 16:22:17.618332 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:17.618306 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf168fd09_9fc9_42bd_9433_79f18e452fd2.slice/crio-81214cd5c3c21bf3329e4404b6cf450f520a49b07315ba91044713c674579d4a WatchSource:0}: Error finding container 81214cd5c3c21bf3329e4404b6cf450f520a49b07315ba91044713c674579d4a: Status 404 returned error can't find the container with id 81214cd5c3c21bf3329e4404b6cf450f520a49b07315ba91044713c674579d4a Sep 08 16:22:18.104815 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.104774 2555 generic.go:356] "Generic (PLEG): container finished" podID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerID="89cb03fbeb51de6f713ea1b60758ac037ab9b0ac0d3f6843fd8205367792de40" exitCode=0 Sep 08 16:22:18.105321 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.104872 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" event={"ID":"f168fd09-9fc9-42bd-9433-79f18e452fd2","Type":"ContainerDied","Data":"89cb03fbeb51de6f713ea1b60758ac037ab9b0ac0d3f6843fd8205367792de40"} Sep 08 16:22:18.105321 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.104912 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" event={"ID":"f168fd09-9fc9-42bd-9433-79f18e452fd2","Type":"ContainerStarted","Data":"81214cd5c3c21bf3329e4404b6cf450f520a49b07315ba91044713c674579d4a"} Sep 08 16:22:18.244033 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.243998 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b"] Sep 08 16:22:18.246618 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.246596 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.249218 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.249189 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 16:22:18.249361 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.249341 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qjv57\"" Sep 08 16:22:18.249423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.249363 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 16:22:18.249423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.249387 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 16:22:18.250194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.250177 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 16:22:18.263609 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.263576 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b"] Sep 08 16:22:18.275796 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.275765 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1882ea1a-62ba-4afb-a324-92d4c65e2146-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.275934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.275805 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1882ea1a-62ba-4afb-a324-92d4c65e2146-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.275934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.275873 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1882ea1a-62ba-4afb-a324-92d4c65e2146-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.275934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.275907 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1882ea1a-62ba-4afb-a324-92d4c65e2146-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.275934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.275930 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tns2\" (UniqueName: \"kubernetes.io/projected/1882ea1a-62ba-4afb-a324-92d4c65e2146-kube-api-access-6tns2\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.377204 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.377118 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1882ea1a-62ba-4afb-a324-92d4c65e2146-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.377204 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.377158 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1882ea1a-62ba-4afb-a324-92d4c65e2146-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.377204 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.377178 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tns2\" (UniqueName: \"kubernetes.io/projected/1882ea1a-62ba-4afb-a324-92d4c65e2146-kube-api-access-6tns2\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.377443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.377223 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1882ea1a-62ba-4afb-a324-92d4c65e2146-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.377443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.377257 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1882ea1a-62ba-4afb-a324-92d4c65e2146-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.377687 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.377651 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1882ea1a-62ba-4afb-a324-92d4c65e2146-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.379624 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.379591 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1882ea1a-62ba-4afb-a324-92d4c65e2146-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.379741 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.379700 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1882ea1a-62ba-4afb-a324-92d4c65e2146-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.379812 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.379766 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1882ea1a-62ba-4afb-a324-92d4c65e2146-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.385599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.385572 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tns2\" (UniqueName: \"kubernetes.io/projected/1882ea1a-62ba-4afb-a324-92d4c65e2146-kube-api-access-6tns2\") pod \"opendatahub-operator-controller-manager-68b89c556d-sr52b\" (UID: \"1882ea1a-62ba-4afb-a324-92d4c65e2146\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.556608 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.556574 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:18.870791 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:18.870766 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b"] Sep 08 16:22:18.872628 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:18.872595 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1882ea1a_62ba_4afb_a324_92d4c65e2146.slice/crio-f7ce8955c2f159776a5d13f974077132a4ecbcdd9cbcdb7fd7705443af8b5eaa WatchSource:0}: Error finding container f7ce8955c2f159776a5d13f974077132a4ecbcdd9cbcdb7fd7705443af8b5eaa: Status 404 returned error can't find the container with id f7ce8955c2f159776a5d13f974077132a4ecbcdd9cbcdb7fd7705443af8b5eaa Sep 08 16:22:19.111133 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:19.111092 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" event={"ID":"1882ea1a-62ba-4afb-a324-92d4c65e2146","Type":"ContainerStarted","Data":"f7ce8955c2f159776a5d13f974077132a4ecbcdd9cbcdb7fd7705443af8b5eaa"} Sep 08 16:22:19.113101 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:19.113072 2555 generic.go:356] "Generic (PLEG): container finished" podID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerID="cbac2d0867b4e2f2786cb5fa4f258979fc8e53f3091224ace9a3fb521e7abde1" exitCode=0 Sep 08 16:22:19.113206 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:19.113153 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" event={"ID":"f168fd09-9fc9-42bd-9433-79f18e452fd2","Type":"ContainerDied","Data":"cbac2d0867b4e2f2786cb5fa4f258979fc8e53f3091224ace9a3fb521e7abde1"} Sep 08 16:22:20.123193 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:20.123032 2555 generic.go:356] "Generic (PLEG): container finished" podID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerID="647094b842a861a92549b2273525cbbc7ca8b987326d23754f1a1cef7622290a" exitCode=0 Sep 08 16:22:20.123193 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:20.123082 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" event={"ID":"f168fd09-9fc9-42bd-9433-79f18e452fd2","Type":"ContainerDied","Data":"647094b842a861a92549b2273525cbbc7ca8b987326d23754f1a1cef7622290a"} Sep 08 16:22:21.899995 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:21.899959 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:21.904021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:21.903999 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crblf\" (UniqueName: \"kubernetes.io/projected/f168fd09-9fc9-42bd-9433-79f18e452fd2-kube-api-access-crblf\") pod \"f168fd09-9fc9-42bd-9433-79f18e452fd2\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " Sep 08 16:22:21.904227 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:21.904036 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-bundle\") pod \"f168fd09-9fc9-42bd-9433-79f18e452fd2\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " Sep 08 16:22:21.904227 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:21.904055 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-util\") pod \"f168fd09-9fc9-42bd-9433-79f18e452fd2\" (UID: \"f168fd09-9fc9-42bd-9433-79f18e452fd2\") " Sep 08 16:22:21.904839 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:21.904815 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-bundle" (OuterVolumeSpecName: "bundle") pod "f168fd09-9fc9-42bd-9433-79f18e452fd2" (UID: "f168fd09-9fc9-42bd-9433-79f18e452fd2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:22:21.906171 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:21.906143 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f168fd09-9fc9-42bd-9433-79f18e452fd2-kube-api-access-crblf" (OuterVolumeSpecName: "kube-api-access-crblf") pod "f168fd09-9fc9-42bd-9433-79f18e452fd2" (UID: "f168fd09-9fc9-42bd-9433-79f18e452fd2"). InnerVolumeSpecName "kube-api-access-crblf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:22:21.909431 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:21.909410 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-util" (OuterVolumeSpecName: "util") pod "f168fd09-9fc9-42bd-9433-79f18e452fd2" (UID: "f168fd09-9fc9-42bd-9433-79f18e452fd2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:22:22.005614 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:22.005570 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crblf\" (UniqueName: \"kubernetes.io/projected/f168fd09-9fc9-42bd-9433-79f18e452fd2-kube-api-access-crblf\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:22:22.005614 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:22.005604 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:22:22.005614 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:22.005614 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f168fd09-9fc9-42bd-9433-79f18e452fd2-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:22:22.136706 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:22.136618 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" event={"ID":"1882ea1a-62ba-4afb-a324-92d4c65e2146","Type":"ContainerStarted","Data":"64fb75d0c2b6e15fe6ca00d0ca5fce902f7e36a2a775bc8cde345aa341558b90"} Sep 08 16:22:22.138583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:22.138563 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" Sep 08 16:22:22.138715 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:22.138600 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r695v" event={"ID":"f168fd09-9fc9-42bd-9433-79f18e452fd2","Type":"ContainerDied","Data":"81214cd5c3c21bf3329e4404b6cf450f520a49b07315ba91044713c674579d4a"} Sep 08 16:22:22.138715 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:22.138641 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81214cd5c3c21bf3329e4404b6cf450f520a49b07315ba91044713c674579d4a" Sep 08 16:22:23.145693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:23.145653 2555 generic.go:356] "Generic (PLEG): container finished" podID="1882ea1a-62ba-4afb-a324-92d4c65e2146" containerID="64fb75d0c2b6e15fe6ca00d0ca5fce902f7e36a2a775bc8cde345aa341558b90" exitCode=0 Sep 08 16:22:23.146103 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:23.145749 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" event={"ID":"1882ea1a-62ba-4afb-a324-92d4c65e2146","Type":"ContainerDied","Data":"64fb75d0c2b6e15fe6ca00d0ca5fce902f7e36a2a775bc8cde345aa341558b90"} Sep 08 16:22:24.154213 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:24.154182 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" event={"ID":"1882ea1a-62ba-4afb-a324-92d4c65e2146","Type":"ContainerStarted","Data":"5d5c34171b2672ef56e3e27a75e74fe0f3da0605a9d466f78f067e0bc1ee9e0f"} Sep 08 16:22:24.154669 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:24.154331 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:24.175912 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:24.175844 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" podStartSLOduration=1.680399006 podStartE2EDuration="6.175823132s" totalImagesPullingTime="3.426945668s" totalInitContainerRuntime="1.068478458s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:24.174515715 +0000 UTC m=+577.769514495" watchObservedRunningTime="2026-09-08 16:22:24.175823132 +0000 UTC m=+577.770821908" Sep 08 16:22:35.162207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:35.162174 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-sr52b" Sep 08 16:22:43.202471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.202430 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b99549648-wpl4m"] Sep 08 16:22:43.203009 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.202987 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerName="extract" Sep 08 16:22:43.203061 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.203013 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerName="extract" Sep 08 16:22:43.203061 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.203027 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerName="pull" Sep 08 16:22:43.203061 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.203037 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerName="pull" Sep 08 16:22:43.203061 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.203050 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerName="util" Sep 08 16:22:43.203061 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.203059 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerName="util" Sep 08 16:22:43.203225 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.203149 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f168fd09-9fc9-42bd-9433-79f18e452fd2" containerName="extract" Sep 08 16:22:43.206171 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.206149 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.208675 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.208636 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 16:22:43.208836 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.208715 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zwsrm\"" Sep 08 16:22:43.208836 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.208643 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 16:22:43.208966 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.208721 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 16:22:43.208966 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.208643 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 16:22:43.216022 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.215985 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b99549648-wpl4m"] Sep 08 16:22:43.283337 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.283296 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl24p\" (UniqueName: \"kubernetes.io/projected/40dde0df-7dc8-4377-bd6f-7362e4c983e3-kube-api-access-wl24p\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.283337 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.283340 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40dde0df-7dc8-4377-bd6f-7362e4c983e3-tmp\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.283549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.283413 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dde0df-7dc8-4377-bd6f-7362e4c983e3-tls-certs\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.384798 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.384757 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dde0df-7dc8-4377-bd6f-7362e4c983e3-tls-certs\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.384964 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.384838 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl24p\" (UniqueName: \"kubernetes.io/projected/40dde0df-7dc8-4377-bd6f-7362e4c983e3-kube-api-access-wl24p\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.384964 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.384861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40dde0df-7dc8-4377-bd6f-7362e4c983e3-tmp\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.387224 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.387198 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40dde0df-7dc8-4377-bd6f-7362e4c983e3-tmp\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.387406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.387362 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dde0df-7dc8-4377-bd6f-7362e4c983e3-tls-certs\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.393203 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.393176 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl24p\" (UniqueName: \"kubernetes.io/projected/40dde0df-7dc8-4377-bd6f-7362e4c983e3-kube-api-access-wl24p\") pod \"kube-auth-proxy-7b99549648-wpl4m\" (UID: \"40dde0df-7dc8-4377-bd6f-7362e4c983e3\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.516778 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.516687 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" Sep 08 16:22:43.803397 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:43.803328 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b99549648-wpl4m"] Sep 08 16:22:43.805600 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:43.805568 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dde0df_7dc8_4377_bd6f_7362e4c983e3.slice/crio-12c42c84eb94f83c09372d77abc652b96540918d21582d109fc48a77be996266 WatchSource:0}: Error finding container 12c42c84eb94f83c09372d77abc652b96540918d21582d109fc48a77be996266: Status 404 returned error can't find the container with id 12c42c84eb94f83c09372d77abc652b96540918d21582d109fc48a77be996266 Sep 08 16:22:44.261956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:44.261906 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" event={"ID":"40dde0df-7dc8-4377-bd6f-7362e4c983e3","Type":"ContainerStarted","Data":"12c42c84eb94f83c09372d77abc652b96540918d21582d109fc48a77be996266"} Sep 08 16:22:47.280196 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:47.280157 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" event={"ID":"40dde0df-7dc8-4377-bd6f-7362e4c983e3","Type":"ContainerStarted","Data":"2192e2bbeea30a334052a836d013c48caff03b667e4d2aa7d40443474cfce8d4"} Sep 08 16:22:47.297401 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:47.297342 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b99549648-wpl4m" podStartSLOduration=1.616940818 podStartE2EDuration="4.297324362s" totalImagesPullingTime="2.680383544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:47.295427133 +0000 UTC m=+600.890425919" watchObservedRunningTime="2026-09-08 16:22:47.297324362 +0000 UTC m=+600.892323131" Sep 08 16:22:48.450027 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.449988 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m5ndb"] Sep 08 16:22:48.453315 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.453292 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:48.455887 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.455859 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 08 16:22:48.456087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.456068 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9kqmn\"" Sep 08 16:22:48.465354 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.464549 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m5ndb"] Sep 08 16:22:48.531800 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.531763 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjp7\" (UniqueName: \"kubernetes.io/projected/3fc8de8d-f903-4533-9b8f-29f992ef2457-kube-api-access-dsjp7\") pod \"odh-model-controller-7998c67dc4-m5ndb\" (UID: \"3fc8de8d-f903-4533-9b8f-29f992ef2457\") " pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:48.531800 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.531815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc8de8d-f903-4533-9b8f-29f992ef2457-cert\") pod \"odh-model-controller-7998c67dc4-m5ndb\" (UID: \"3fc8de8d-f903-4533-9b8f-29f992ef2457\") " pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:48.632595 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.632542 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjp7\" (UniqueName: \"kubernetes.io/projected/3fc8de8d-f903-4533-9b8f-29f992ef2457-kube-api-access-dsjp7\") pod \"odh-model-controller-7998c67dc4-m5ndb\" (UID: \"3fc8de8d-f903-4533-9b8f-29f992ef2457\") " pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:48.632595 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.632597 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc8de8d-f903-4533-9b8f-29f992ef2457-cert\") pod \"odh-model-controller-7998c67dc4-m5ndb\" (UID: \"3fc8de8d-f903-4533-9b8f-29f992ef2457\") " pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:48.635094 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.635072 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc8de8d-f903-4533-9b8f-29f992ef2457-cert\") pod \"odh-model-controller-7998c67dc4-m5ndb\" (UID: \"3fc8de8d-f903-4533-9b8f-29f992ef2457\") " pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:48.641150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.641121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjp7\" (UniqueName: \"kubernetes.io/projected/3fc8de8d-f903-4533-9b8f-29f992ef2457-kube-api-access-dsjp7\") pod \"odh-model-controller-7998c67dc4-m5ndb\" (UID: \"3fc8de8d-f903-4533-9b8f-29f992ef2457\") " pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:48.765966 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:48.765857 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:49.000732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.000688 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xzwnj"] Sep 08 16:22:49.004999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.004977 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.008435 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.008410 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 08 16:22:49.026933 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.026848 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xzwnj"] Sep 08 16:22:49.060535 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.060503 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-m5ndb"] Sep 08 16:22:49.061886 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:49.061858 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc8de8d_f903_4533_9b8f_29f992ef2457.slice/crio-7d331d0e2c5ffd146304d0e8c60741a98bf179ee28a68cfda8f3ecb04ae0bfbb WatchSource:0}: Error finding container 7d331d0e2c5ffd146304d0e8c60741a98bf179ee28a68cfda8f3ecb04ae0bfbb: Status 404 returned error can't find the container with id 7d331d0e2c5ffd146304d0e8c60741a98bf179ee28a68cfda8f3ecb04ae0bfbb Sep 08 16:22:49.135894 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.135854 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d2d37b-1ad6-4901-878c-0d3bd81a9761-tls-certs\") pod \"model-serving-api-54d89bd5fc-xzwnj\" (UID: \"85d2d37b-1ad6-4901-878c-0d3bd81a9761\") " pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.135894 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.135890 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgnq\" (UniqueName: \"kubernetes.io/projected/85d2d37b-1ad6-4901-878c-0d3bd81a9761-kube-api-access-2cgnq\") pod \"model-serving-api-54d89bd5fc-xzwnj\" (UID: \"85d2d37b-1ad6-4901-878c-0d3bd81a9761\") " pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.236573 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.236532 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d2d37b-1ad6-4901-878c-0d3bd81a9761-tls-certs\") pod \"model-serving-api-54d89bd5fc-xzwnj\" (UID: \"85d2d37b-1ad6-4901-878c-0d3bd81a9761\") " pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.236573 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.236576 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgnq\" (UniqueName: \"kubernetes.io/projected/85d2d37b-1ad6-4901-878c-0d3bd81a9761-kube-api-access-2cgnq\") pod \"model-serving-api-54d89bd5fc-xzwnj\" (UID: \"85d2d37b-1ad6-4901-878c-0d3bd81a9761\") " pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.239055 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.239022 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d2d37b-1ad6-4901-878c-0d3bd81a9761-tls-certs\") pod \"model-serving-api-54d89bd5fc-xzwnj\" (UID: \"85d2d37b-1ad6-4901-878c-0d3bd81a9761\") " pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.247133 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.247089 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgnq\" (UniqueName: \"kubernetes.io/projected/85d2d37b-1ad6-4901-878c-0d3bd81a9761-kube-api-access-2cgnq\") pod \"model-serving-api-54d89bd5fc-xzwnj\" (UID: \"85d2d37b-1ad6-4901-878c-0d3bd81a9761\") " pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.293703 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.293610 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" event={"ID":"3fc8de8d-f903-4533-9b8f-29f992ef2457","Type":"ContainerStarted","Data":"7d331d0e2c5ffd146304d0e8c60741a98bf179ee28a68cfda8f3ecb04ae0bfbb"} Sep 08 16:22:49.315351 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.315308 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:49.475210 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.475171 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl"] Sep 08 16:22:49.480259 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.480222 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.482824 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.482787 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 16:22:49.483279 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.483256 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8lpp6\"" Sep 08 16:22:49.493188 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.493156 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl"] Sep 08 16:22:49.539374 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539341 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539386 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539413 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539439 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539455 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539473 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539536 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrcd8\" (UniqueName: \"kubernetes.io/projected/538b9d79-d7f5-40df-b1b8-3ead45bd262d-kube-api-access-vrcd8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.539754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.539646 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.592554 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.592520 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xzwnj"] Sep 08 16:22:49.594497 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:49.594470 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d2d37b_1ad6_4901_878c_0d3bd81a9761.slice/crio-1f0669a2bc574d797ccad71df782acc9d2c5d49f46defea67b5c1e00f57a5306 WatchSource:0}: Error finding container 1f0669a2bc574d797ccad71df782acc9d2c5d49f46defea67b5c1e00f57a5306: Status 404 returned error can't find the container with id 1f0669a2bc574d797ccad71df782acc9d2c5d49f46defea67b5c1e00f57a5306 Sep 08 16:22:49.640602 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640564 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.640812 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640612 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.640812 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640739 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.640941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640835 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.640941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640866 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.640941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640909 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641086 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640960 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641086 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.640984 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrcd8\" (UniqueName: \"kubernetes.io/projected/538b9d79-d7f5-40df-b1b8-3ead45bd262d-kube-api-access-vrcd8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641195 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.641083 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.641301 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.641444 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.641739 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641891 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.641851 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.641947 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.641928 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.643502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.643474 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.643639 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.643621 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.649980 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.649953 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/538b9d79-d7f5-40df-b1b8-3ead45bd262d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.650200 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.650181 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrcd8\" (UniqueName: \"kubernetes.io/projected/538b9d79-d7f5-40df-b1b8-3ead45bd262d-kube-api-access-vrcd8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl\" (UID: \"538b9d79-d7f5-40df-b1b8-3ead45bd262d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:49.771182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.771145 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c76b748c5-gs8mw"] Sep 08 16:22:49.795410 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:49.795372 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:50.074723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:50.074683 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl"] Sep 08 16:22:50.076533 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:50.076502 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538b9d79_d7f5_40df_b1b8_3ead45bd262d.slice/crio-22ef841bd244a9ce2356986d365dfb2708f3022d257341ff97b241a2da83e096 WatchSource:0}: Error finding container 22ef841bd244a9ce2356986d365dfb2708f3022d257341ff97b241a2da83e096: Status 404 returned error can't find the container with id 22ef841bd244a9ce2356986d365dfb2708f3022d257341ff97b241a2da83e096 Sep 08 16:22:50.301877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:50.301785 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" event={"ID":"538b9d79-d7f5-40df-b1b8-3ead45bd262d","Type":"ContainerStarted","Data":"22ef841bd244a9ce2356986d365dfb2708f3022d257341ff97b241a2da83e096"} Sep 08 16:22:50.303297 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:50.303237 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" event={"ID":"85d2d37b-1ad6-4901-878c-0d3bd81a9761","Type":"ContainerStarted","Data":"1f0669a2bc574d797ccad71df782acc9d2c5d49f46defea67b5c1e00f57a5306"} Sep 08 16:22:53.326289 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:53.326175 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" event={"ID":"3fc8de8d-f903-4533-9b8f-29f992ef2457","Type":"ContainerStarted","Data":"b6fbfa599c0b1a5efd58a26a199c635ce5ad3b30cf94ed518ccd805b4e8dd489"} Sep 08 16:22:53.326813 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:53.326305 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:22:53.328602 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:53.328574 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" event={"ID":"85d2d37b-1ad6-4901-878c-0d3bd81a9761","Type":"ContainerStarted","Data":"cf55a2ec21f5f55071063dd66c0217b0b6776590dfdc4b12b124bfcef1a922d8"} Sep 08 16:22:53.328741 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:53.328726 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:22:53.345227 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:53.344679 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" podStartSLOduration=1.4825004640000001 podStartE2EDuration="5.344654487s" totalImagesPullingTime="3.862154023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:53.341778657 +0000 UTC m=+606.936777442" watchObservedRunningTime="2026-09-08 16:22:53.344654487 +0000 UTC m=+606.939653324" Sep 08 16:22:53.360543 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:53.360017 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" podStartSLOduration=2.093247604 podStartE2EDuration="5.359994036s" totalImagesPullingTime="3.266746432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:53.357551349 +0000 UTC m=+606.952550132" watchObservedRunningTime="2026-09-08 16:22:53.359994036 +0000 UTC m=+606.954992796" Sep 08 16:22:54.479309 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:54.479264 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:22:54.479717 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:54.479364 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:22:54.479717 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:54.479415 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:22:55.340182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:55.340140 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" event={"ID":"538b9d79-d7f5-40df-b1b8-3ead45bd262d","Type":"ContainerStarted","Data":"23731eb4550e5be4436767bdf2110c4190f2e20ded7b9df353bb1f1ed430b700"} Sep 08 16:22:55.359830 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:55.359772 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" podStartSLOduration=1.959895033 podStartE2EDuration="6.359752715s" totalImagesPullingTime="4.399857682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:55.358025047 +0000 UTC m=+608.953023806" watchObservedRunningTime="2026-09-08 16:22:55.359752715 +0000 UTC m=+608.954751475" Sep 08 16:22:55.795812 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:55.795764 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:55.800611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:55.800582 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:56.346705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.346669 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:56.347691 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.347659 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl" Sep 08 16:22:56.494103 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.494064 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78fd8475-jmng8"] Sep 08 16:22:56.496753 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.496730 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.499576 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.499554 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zq85p\"" Sep 08 16:22:56.499693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.499555 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 08 16:22:56.499693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.499680 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 08 16:22:56.499786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.499737 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 08 16:22:56.508853 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.508818 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78fd8475-jmng8"] Sep 08 16:22:56.613493 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.613390 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-manager-config\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.613493 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.613450 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-cert\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.613677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.613512 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zmp\" (UniqueName: \"kubernetes.io/projected/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-kube-api-access-q6zmp\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.613677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.613618 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-metrics-cert\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.715065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.715026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-metrics-cert\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.715269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.715124 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-manager-config\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.715269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.715170 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-cert\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.715269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.715197 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zmp\" (UniqueName: \"kubernetes.io/projected/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-kube-api-access-q6zmp\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.715898 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.715872 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-manager-config\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.717653 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.717632 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-metrics-cert\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.717792 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.717772 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-cert\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.730830 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.730802 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zmp\" (UniqueName: \"kubernetes.io/projected/a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566-kube-api-access-q6zmp\") pod \"lws-controller-manager-78fd8475-jmng8\" (UID: \"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566\") " pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:56.807353 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:56.807308 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:57.089441 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:57.089357 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78fd8475-jmng8"] Sep 08 16:22:57.092725 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:22:57.092695 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a38ecf_bcd6_4d6d_a370_da4eb7cd7566.slice/crio-e3ca3874b3f5dba83d9a60c5b5adafaec3e407013b21baabff57ab685cdf4395 WatchSource:0}: Error finding container e3ca3874b3f5dba83d9a60c5b5adafaec3e407013b21baabff57ab685cdf4395: Status 404 returned error can't find the container with id e3ca3874b3f5dba83d9a60c5b5adafaec3e407013b21baabff57ab685cdf4395 Sep 08 16:22:57.352921 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:57.352882 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" event={"ID":"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566","Type":"ContainerStarted","Data":"e3ca3874b3f5dba83d9a60c5b5adafaec3e407013b21baabff57ab685cdf4395"} Sep 08 16:22:59.370050 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:59.370014 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" event={"ID":"a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566","Type":"ContainerStarted","Data":"8238514d1e5e0081da98979e323767bde6734d139c9633f56de14622d35d77e6"} Sep 08 16:22:59.370497 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:59.370086 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:22:59.392344 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:22:59.392273 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" podStartSLOduration=1.81681574 podStartE2EDuration="3.392255945s" totalImagesPullingTime="1.575440205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:59.390703592 +0000 UTC m=+612.985702353" watchObservedRunningTime="2026-09-08 16:22:59.392255945 +0000 UTC m=+612.987254768" Sep 08 16:23:04.336174 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:04.336140 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-m5ndb" Sep 08 16:23:04.338220 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:04.338199 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xzwnj" Sep 08 16:23:10.377714 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:10.377629 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78fd8475-jmng8" Sep 08 16:23:14.793722 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:14.793678 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c76b748c5-gs8mw" podUID="22b4d105-b934-4f48-ae54-7505be206cd7" containerName="console" containerID="cri-o://087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73" gracePeriod=15 Sep 08 16:23:15.043653 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.043622 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:23:15.082648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.082565 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-oauth-config\") pod \"22b4d105-b934-4f48-ae54-7505be206cd7\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " Sep 08 16:23:15.082648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.082622 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qdck\" (UniqueName: \"kubernetes.io/projected/22b4d105-b934-4f48-ae54-7505be206cd7-kube-api-access-7qdck\") pod \"22b4d105-b934-4f48-ae54-7505be206cd7\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " Sep 08 16:23:15.082877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.082675 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-service-ca\") pod \"22b4d105-b934-4f48-ae54-7505be206cd7\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " Sep 08 16:23:15.082877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.082778 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-console-config\") pod \"22b4d105-b934-4f48-ae54-7505be206cd7\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " Sep 08 16:23:15.082877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.082851 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-oauth-serving-cert\") pod \"22b4d105-b934-4f48-ae54-7505be206cd7\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " Sep 08 16:23:15.083035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.082892 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-serving-cert\") pod \"22b4d105-b934-4f48-ae54-7505be206cd7\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " Sep 08 16:23:15.083035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.082920 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-trusted-ca-bundle\") pod \"22b4d105-b934-4f48-ae54-7505be206cd7\" (UID: \"22b4d105-b934-4f48-ae54-7505be206cd7\") " Sep 08 16:23:15.083488 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.083312 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "22b4d105-b934-4f48-ae54-7505be206cd7" (UID: "22b4d105-b934-4f48-ae54-7505be206cd7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:23:15.083614 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.083496 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-console-config" (OuterVolumeSpecName: "console-config") pod "22b4d105-b934-4f48-ae54-7505be206cd7" (UID: "22b4d105-b934-4f48-ae54-7505be206cd7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:23:15.083614 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.083562 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "22b4d105-b934-4f48-ae54-7505be206cd7" (UID: "22b4d105-b934-4f48-ae54-7505be206cd7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:23:15.083739 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.083635 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-service-ca" (OuterVolumeSpecName: "service-ca") pod "22b4d105-b934-4f48-ae54-7505be206cd7" (UID: "22b4d105-b934-4f48-ae54-7505be206cd7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:23:15.085038 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.085003 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b4d105-b934-4f48-ae54-7505be206cd7-kube-api-access-7qdck" (OuterVolumeSpecName: "kube-api-access-7qdck") pod "22b4d105-b934-4f48-ae54-7505be206cd7" (UID: "22b4d105-b934-4f48-ae54-7505be206cd7"). InnerVolumeSpecName "kube-api-access-7qdck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:23:15.085138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.085102 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "22b4d105-b934-4f48-ae54-7505be206cd7" (UID: "22b4d105-b934-4f48-ae54-7505be206cd7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:23:15.085211 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.085186 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "22b4d105-b934-4f48-ae54-7505be206cd7" (UID: "22b4d105-b934-4f48-ae54-7505be206cd7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:23:15.184470 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.184429 2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:15.184470 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.184462 2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-console-config\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:15.184470 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.184471 2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-oauth-serving-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:15.184470 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.184480 2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-serving-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:15.184751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.184489 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b4d105-b934-4f48-ae54-7505be206cd7-trusted-ca-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:15.184751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.184497 2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/22b4d105-b934-4f48-ae54-7505be206cd7-console-oauth-config\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:15.184751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.184505 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdck\" (UniqueName: \"kubernetes.io/projected/22b4d105-b934-4f48-ae54-7505be206cd7-kube-api-access-7qdck\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:15.452932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.452898 2555 generic.go:356] "Generic (PLEG): container finished" podID="22b4d105-b934-4f48-ae54-7505be206cd7" containerID="087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73" exitCode=0 Sep 08 16:23:15.453126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.452946 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c76b748c5-gs8mw" Sep 08 16:23:15.453126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.452981 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c76b748c5-gs8mw" event={"ID":"22b4d105-b934-4f48-ae54-7505be206cd7","Type":"ContainerDied","Data":"087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73"} Sep 08 16:23:15.453126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.453031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c76b748c5-gs8mw" event={"ID":"22b4d105-b934-4f48-ae54-7505be206cd7","Type":"ContainerDied","Data":"0038f17f021b8f0708b82bebb0bb2a0761618bbb5e4336964bf859da47245632"} Sep 08 16:23:15.453126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.453055 2555 scope.go:117] "RemoveContainer" containerID="087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73" Sep 08 16:23:15.463545 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.463520 2555 scope.go:117] "RemoveContainer" containerID="087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73" Sep 08 16:23:15.463909 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:23:15.463885 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73\": container with ID starting with 087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73 not found: ID does not exist" containerID="087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73" Sep 08 16:23:15.463961 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.463921 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73"} err="failed to get container status \"087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73\": rpc error: code = NotFound desc = could not find container \"087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73\": container with ID starting with 087c6a6888f128dcad3ea579b02f0554024ae329b2a54840bd1137ee1396ff73 not found: ID does not exist" Sep 08 16:23:15.476114 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.476074 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c76b748c5-gs8mw"] Sep 08 16:23:15.481762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:15.481729 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c76b748c5-gs8mw"] Sep 08 16:23:16.937199 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:16.937163 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b4d105-b934-4f48-ae54-7505be206cd7" path="/var/lib/kubelet/pods/22b4d105-b934-4f48-ae54-7505be206cd7/volumes" Sep 08 16:23:53.855631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.855588 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz"] Sep 08 16:23:53.856219 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.856123 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b4d105-b934-4f48-ae54-7505be206cd7" containerName="console" Sep 08 16:23:53.856219 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.856143 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b4d105-b934-4f48-ae54-7505be206cd7" containerName="console" Sep 08 16:23:53.856394 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.856235 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b4d105-b934-4f48-ae54-7505be206cd7" containerName="console" Sep 08 16:23:53.859307 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.859286 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:53.861899 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.861876 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 16:23:53.862797 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.862780 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9vsgj\"" Sep 08 16:23:53.862894 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.862792 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 16:23:53.866317 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.866292 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz"] Sep 08 16:23:53.958259 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.958205 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7"] Sep 08 16:23:53.960843 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.960825 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:53.970086 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:53.970058 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7"] Sep 08 16:23:54.020852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.020809 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.020852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.020855 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5zc\" (UniqueName: \"kubernetes.io/projected/0a801e9c-1765-423a-acc4-798004ac1f93-kube-api-access-rg5zc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.021083 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.020946 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.053412 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.053377 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh"] Sep 08 16:23:54.056315 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.056298 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.065394 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.065365 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh"] Sep 08 16:23:54.122221 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122126 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.122221 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122188 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.122429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122276 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.122429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122357 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6lt\" (UniqueName: \"kubernetes.io/projected/1951dbc6-f924-4a26-bcc8-78765468652b-kube-api-access-gg6lt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.122429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122398 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.122429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122419 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5zc\" (UniqueName: \"kubernetes.io/projected/0a801e9c-1765-423a-acc4-798004ac1f93-kube-api-access-rg5zc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.122607 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122586 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.122688 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.122673 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.130723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.130684 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5zc\" (UniqueName: \"kubernetes.io/projected/0a801e9c-1765-423a-acc4-798004ac1f93-kube-api-access-rg5zc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.160784 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.160749 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh"] Sep 08 16:23:54.163888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.163870 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.169392 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.169371 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:54.173896 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.173867 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh"] Sep 08 16:23:54.223821 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.223785 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdg97\" (UniqueName: \"kubernetes.io/projected/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-kube-api-access-vdg97\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.223965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.223838 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.223965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.223924 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.224079 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.224026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6lt\" (UniqueName: \"kubernetes.io/projected/1951dbc6-f924-4a26-bcc8-78765468652b-kube-api-access-gg6lt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.224079 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.224060 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.224185 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.224131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.224438 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.224394 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.224542 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.224518 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.233463 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.233431 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6lt\" (UniqueName: \"kubernetes.io/projected/1951dbc6-f924-4a26-bcc8-78765468652b-kube-api-access-gg6lt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.271409 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.271370 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:54.325586 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.325553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.325680 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.325619 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4wm\" (UniqueName: \"kubernetes.io/projected/be7c8429-b684-4d1f-9ed3-521a3acf11f4-kube-api-access-5r4wm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.325680 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.325663 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdg97\" (UniqueName: \"kubernetes.io/projected/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-kube-api-access-vdg97\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.325796 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.325695 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.325796 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.325736 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.325900 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.325800 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.326599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.326234 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.326599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.326533 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.336343 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.336314 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdg97\" (UniqueName: \"kubernetes.io/projected/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-kube-api-access-vdg97\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.365982 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.365939 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:54.427308 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.427109 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.427308 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.427303 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4wm\" (UniqueName: \"kubernetes.io/projected/be7c8429-b684-4d1f-9ed3-521a3acf11f4-kube-api-access-5r4wm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.427576 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.427374 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.427576 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.427500 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.427839 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.427816 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.436179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.436153 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4wm\" (UniqueName: \"kubernetes.io/projected/be7c8429-b684-4d1f-9ed3-521a3acf11f4-kube-api-access-5r4wm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.445662 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.445506 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz"] Sep 08 16:23:54.448770 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:23:54.448740 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a801e9c_1765_423a_acc4_798004ac1f93.slice/crio-820992c70180a637724a3fabb7056ff3e33bbe3917a8866b1d372587ddef3e87 WatchSource:0}: Error finding container 820992c70180a637724a3fabb7056ff3e33bbe3917a8866b1d372587ddef3e87: Status 404 returned error can't find the container with id 820992c70180a637724a3fabb7056ff3e33bbe3917a8866b1d372587ddef3e87 Sep 08 16:23:54.474403 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.474206 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:54.550373 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.550347 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7"] Sep 08 16:23:54.591157 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:23:54.591121 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1951dbc6_f924_4a26_bcc8_78765468652b.slice/crio-4d2c27fe3137c72df41aafd14ff05dd3a0b43b399aea7a76fc441595bac3245a WatchSource:0}: Error finding container 4d2c27fe3137c72df41aafd14ff05dd3a0b43b399aea7a76fc441595bac3245a: Status 404 returned error can't find the container with id 4d2c27fe3137c72df41aafd14ff05dd3a0b43b399aea7a76fc441595bac3245a Sep 08 16:23:54.639203 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.639177 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh"] Sep 08 16:23:54.641042 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:23:54.641017 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32970005_f9e5_4fc5_a7c3_f75a7ddf786a.slice/crio-29a037275d832690d36bfca1abe07535b10acf0b8a51f22741d1f1250739d4a2 WatchSource:0}: Error finding container 29a037275d832690d36bfca1abe07535b10acf0b8a51f22741d1f1250739d4a2: Status 404 returned error can't find the container with id 29a037275d832690d36bfca1abe07535b10acf0b8a51f22741d1f1250739d4a2 Sep 08 16:23:54.654509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.654474 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" event={"ID":"32970005-f9e5-4fc5-a7c3-f75a7ddf786a","Type":"ContainerStarted","Data":"29a037275d832690d36bfca1abe07535b10acf0b8a51f22741d1f1250739d4a2"} Sep 08 16:23:54.656481 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.656332 2555 generic.go:356] "Generic (PLEG): container finished" podID="0a801e9c-1765-423a-acc4-798004ac1f93" containerID="5bcf90caaaa7101bb17adcee6209a515d002795e62845c3a21405c535b85661e" exitCode=0 Sep 08 16:23:54.656481 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.656438 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" event={"ID":"0a801e9c-1765-423a-acc4-798004ac1f93","Type":"ContainerDied","Data":"5bcf90caaaa7101bb17adcee6209a515d002795e62845c3a21405c535b85661e"} Sep 08 16:23:54.656640 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.656506 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" event={"ID":"0a801e9c-1765-423a-acc4-798004ac1f93","Type":"ContainerStarted","Data":"820992c70180a637724a3fabb7056ff3e33bbe3917a8866b1d372587ddef3e87"} Sep 08 16:23:54.657633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.657610 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:23:54.658282 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.658260 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" event={"ID":"1951dbc6-f924-4a26-bcc8-78765468652b","Type":"ContainerStarted","Data":"4d2c27fe3137c72df41aafd14ff05dd3a0b43b399aea7a76fc441595bac3245a"} Sep 08 16:23:54.729337 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:23:54.729297 2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1951dbc6_f924_4a26_bcc8_78765468652b.slice/crio-757ae851f011887c5573e7cfc52eeb93118620fbbc36b91c59d723d7b8bdb634.scope\": RecentStats: unable to find data in memory cache]" Sep 08 16:23:54.751424 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:54.751395 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh"] Sep 08 16:23:54.798315 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:23:54.798276 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7c8429_b684_4d1f_9ed3_521a3acf11f4.slice/crio-8f874585ddd1997b20e68d7f3bfff925d1970f005eb9a4f579bc1357952cc9a5 WatchSource:0}: Error finding container 8f874585ddd1997b20e68d7f3bfff925d1970f005eb9a4f579bc1357952cc9a5: Status 404 returned error can't find the container with id 8f874585ddd1997b20e68d7f3bfff925d1970f005eb9a4f579bc1357952cc9a5 Sep 08 16:23:55.666210 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.666174 2555 generic.go:356] "Generic (PLEG): container finished" podID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerID="d25877d9e81d48f87b7a2d92280c0280d0c209bc3abdbe4ee13e3b127153f816" exitCode=0 Sep 08 16:23:55.666690 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.666266 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" event={"ID":"be7c8429-b684-4d1f-9ed3-521a3acf11f4","Type":"ContainerDied","Data":"d25877d9e81d48f87b7a2d92280c0280d0c209bc3abdbe4ee13e3b127153f816"} Sep 08 16:23:55.666690 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.666321 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" event={"ID":"be7c8429-b684-4d1f-9ed3-521a3acf11f4","Type":"ContainerStarted","Data":"8f874585ddd1997b20e68d7f3bfff925d1970f005eb9a4f579bc1357952cc9a5"} Sep 08 16:23:55.667967 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.667942 2555 generic.go:356] "Generic (PLEG): container finished" podID="1951dbc6-f924-4a26-bcc8-78765468652b" containerID="757ae851f011887c5573e7cfc52eeb93118620fbbc36b91c59d723d7b8bdb634" exitCode=0 Sep 08 16:23:55.668056 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.668030 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" event={"ID":"1951dbc6-f924-4a26-bcc8-78765468652b","Type":"ContainerDied","Data":"757ae851f011887c5573e7cfc52eeb93118620fbbc36b91c59d723d7b8bdb634"} Sep 08 16:23:55.669861 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.669840 2555 generic.go:356] "Generic (PLEG): container finished" podID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerID="a076846daf5a14faf79e93a89fcbd221059cdb97951a2972d3c4c4945680d391" exitCode=0 Sep 08 16:23:55.669959 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.669917 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" event={"ID":"32970005-f9e5-4fc5-a7c3-f75a7ddf786a","Type":"ContainerDied","Data":"a076846daf5a14faf79e93a89fcbd221059cdb97951a2972d3c4c4945680d391"} Sep 08 16:23:55.672069 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.672039 2555 generic.go:356] "Generic (PLEG): container finished" podID="0a801e9c-1765-423a-acc4-798004ac1f93" containerID="8098e85dd8587a831dcc67fa41db22a42ea1bf3a00de3358255a65519189265d" exitCode=0 Sep 08 16:23:55.672136 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:55.672072 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" event={"ID":"0a801e9c-1765-423a-acc4-798004ac1f93","Type":"ContainerDied","Data":"8098e85dd8587a831dcc67fa41db22a42ea1bf3a00de3358255a65519189265d"} Sep 08 16:23:56.683760 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:56.683647 2555 generic.go:356] "Generic (PLEG): container finished" podID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerID="4e24ae4421095b6708bb8a19c43f3190125931101c3f1c3e730d4ea17c238d13" exitCode=0 Sep 08 16:23:56.684238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:56.683801 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" event={"ID":"be7c8429-b684-4d1f-9ed3-521a3acf11f4","Type":"ContainerDied","Data":"4e24ae4421095b6708bb8a19c43f3190125931101c3f1c3e730d4ea17c238d13"} Sep 08 16:23:56.686091 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:56.686066 2555 generic.go:356] "Generic (PLEG): container finished" podID="1951dbc6-f924-4a26-bcc8-78765468652b" containerID="5afe732fcc9e8aeddd152ce388e41614cb08d683dc64735336064ec46f5cce3e" exitCode=0 Sep 08 16:23:56.686192 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:56.686145 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" event={"ID":"1951dbc6-f924-4a26-bcc8-78765468652b","Type":"ContainerDied","Data":"5afe732fcc9e8aeddd152ce388e41614cb08d683dc64735336064ec46f5cce3e"} Sep 08 16:23:56.688529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:56.688507 2555 generic.go:356] "Generic (PLEG): container finished" podID="0a801e9c-1765-423a-acc4-798004ac1f93" containerID="08aeed8e3f9640e06aeb95fa3fd3c212a1841e066f78d455e76e5417e51b734c" exitCode=0 Sep 08 16:23:56.688637 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:56.688543 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" event={"ID":"0a801e9c-1765-423a-acc4-798004ac1f93","Type":"ContainerDied","Data":"08aeed8e3f9640e06aeb95fa3fd3c212a1841e066f78d455e76e5417e51b734c"} Sep 08 16:23:57.696150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:57.696116 2555 generic.go:356] "Generic (PLEG): container finished" podID="1951dbc6-f924-4a26-bcc8-78765468652b" containerID="b5c9fb2ed7d6dbf2a489f9f94980175cc713d6d999a7c23087bd78a0a097e24a" exitCode=0 Sep 08 16:23:57.696622 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:57.696193 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" event={"ID":"1951dbc6-f924-4a26-bcc8-78765468652b","Type":"ContainerDied","Data":"b5c9fb2ed7d6dbf2a489f9f94980175cc713d6d999a7c23087bd78a0a097e24a"} Sep 08 16:23:57.701388 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:57.701355 2555 generic.go:356] "Generic (PLEG): container finished" podID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerID="92f7456a34fd77c291e5fe8d1eb6477da23c9da5a1b8af14eccf4acb0e0ca03a" exitCode=0 Sep 08 16:23:57.701547 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:57.701432 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" event={"ID":"32970005-f9e5-4fc5-a7c3-f75a7ddf786a","Type":"ContainerDied","Data":"92f7456a34fd77c291e5fe8d1eb6477da23c9da5a1b8af14eccf4acb0e0ca03a"} Sep 08 16:23:57.703785 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:57.703761 2555 generic.go:356] "Generic (PLEG): container finished" podID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerID="58992f34888ead611419bf042e6542598c1c174122510abc7bc35e1817e1fca9" exitCode=0 Sep 08 16:23:57.703885 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:57.703843 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" event={"ID":"be7c8429-b684-4d1f-9ed3-521a3acf11f4","Type":"ContainerDied","Data":"58992f34888ead611419bf042e6542598c1c174122510abc7bc35e1817e1fca9"} Sep 08 16:23:57.905150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:57.905123 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:58.063126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.063025 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg5zc\" (UniqueName: \"kubernetes.io/projected/0a801e9c-1765-423a-acc4-798004ac1f93-kube-api-access-rg5zc\") pod \"0a801e9c-1765-423a-acc4-798004ac1f93\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " Sep 08 16:23:58.063341 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.063133 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-util\") pod \"0a801e9c-1765-423a-acc4-798004ac1f93\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " Sep 08 16:23:58.063341 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.063157 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-bundle\") pod \"0a801e9c-1765-423a-acc4-798004ac1f93\" (UID: \"0a801e9c-1765-423a-acc4-798004ac1f93\") " Sep 08 16:23:58.063790 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.063768 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-bundle" (OuterVolumeSpecName: "bundle") pod "0a801e9c-1765-423a-acc4-798004ac1f93" (UID: "0a801e9c-1765-423a-acc4-798004ac1f93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:58.065133 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.065101 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a801e9c-1765-423a-acc4-798004ac1f93-kube-api-access-rg5zc" (OuterVolumeSpecName: "kube-api-access-rg5zc") pod "0a801e9c-1765-423a-acc4-798004ac1f93" (UID: "0a801e9c-1765-423a-acc4-798004ac1f93"). InnerVolumeSpecName "kube-api-access-rg5zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:23:58.068955 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.068908 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-util" (OuterVolumeSpecName: "util") pod "0a801e9c-1765-423a-acc4-798004ac1f93" (UID: "0a801e9c-1765-423a-acc4-798004ac1f93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:58.164701 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.164663 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:58.164701 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.164697 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a801e9c-1765-423a-acc4-798004ac1f93-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:58.164701 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.164708 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg5zc\" (UniqueName: \"kubernetes.io/projected/0a801e9c-1765-423a-acc4-798004ac1f93-kube-api-access-rg5zc\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:58.711296 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.711225 2555 generic.go:356] "Generic (PLEG): container finished" podID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerID="93830494b292705b543a65f14ff7d1b48584b5abe79304e36157fe1dd9fc2648" exitCode=0 Sep 08 16:23:58.711296 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.711287 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" event={"ID":"32970005-f9e5-4fc5-a7c3-f75a7ddf786a","Type":"ContainerDied","Data":"93830494b292705b543a65f14ff7d1b48584b5abe79304e36157fe1dd9fc2648"} Sep 08 16:23:58.713110 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.713091 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" Sep 08 16:23:58.713229 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.713129 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arnsnz" event={"ID":"0a801e9c-1765-423a-acc4-798004ac1f93","Type":"ContainerDied","Data":"820992c70180a637724a3fabb7056ff3e33bbe3917a8866b1d372587ddef3e87"} Sep 08 16:23:58.713229 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.713168 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="820992c70180a637724a3fabb7056ff3e33bbe3917a8866b1d372587ddef3e87" Sep 08 16:23:58.852354 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.852328 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:58.875347 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.875308 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r4wm\" (UniqueName: \"kubernetes.io/projected/be7c8429-b684-4d1f-9ed3-521a3acf11f4-kube-api-access-5r4wm\") pod \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " Sep 08 16:23:58.875527 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.875403 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-util\") pod \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " Sep 08 16:23:58.875527 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.875468 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-bundle\") pod \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\" (UID: \"be7c8429-b684-4d1f-9ed3-521a3acf11f4\") " Sep 08 16:23:58.876121 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.876084 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-bundle" (OuterVolumeSpecName: "bundle") pod "be7c8429-b684-4d1f-9ed3-521a3acf11f4" (UID: "be7c8429-b684-4d1f-9ed3-521a3acf11f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:58.877677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.877652 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7c8429-b684-4d1f-9ed3-521a3acf11f4-kube-api-access-5r4wm" (OuterVolumeSpecName: "kube-api-access-5r4wm") pod "be7c8429-b684-4d1f-9ed3-521a3acf11f4" (UID: "be7c8429-b684-4d1f-9ed3-521a3acf11f4"). InnerVolumeSpecName "kube-api-access-5r4wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:23:58.881112 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.881085 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-util" (OuterVolumeSpecName: "util") pod "be7c8429-b684-4d1f-9ed3-521a3acf11f4" (UID: "be7c8429-b684-4d1f-9ed3-521a3acf11f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:58.884485 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.884464 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:58.976974 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.976870 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg6lt\" (UniqueName: \"kubernetes.io/projected/1951dbc6-f924-4a26-bcc8-78765468652b-kube-api-access-gg6lt\") pod \"1951dbc6-f924-4a26-bcc8-78765468652b\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " Sep 08 16:23:58.976974 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.976939 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-bundle\") pod \"1951dbc6-f924-4a26-bcc8-78765468652b\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " Sep 08 16:23:58.976974 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.976973 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-util\") pod \"1951dbc6-f924-4a26-bcc8-78765468652b\" (UID: \"1951dbc6-f924-4a26-bcc8-78765468652b\") " Sep 08 16:23:58.977281 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.977188 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:58.977281 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.977208 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r4wm\" (UniqueName: \"kubernetes.io/projected/be7c8429-b684-4d1f-9ed3-521a3acf11f4-kube-api-access-5r4wm\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:58.977281 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.977223 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be7c8429-b684-4d1f-9ed3-521a3acf11f4-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:58.977602 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.977575 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-bundle" (OuterVolumeSpecName: "bundle") pod "1951dbc6-f924-4a26-bcc8-78765468652b" (UID: "1951dbc6-f924-4a26-bcc8-78765468652b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:58.979190 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.979166 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1951dbc6-f924-4a26-bcc8-78765468652b-kube-api-access-gg6lt" (OuterVolumeSpecName: "kube-api-access-gg6lt") pod "1951dbc6-f924-4a26-bcc8-78765468652b" (UID: "1951dbc6-f924-4a26-bcc8-78765468652b"). InnerVolumeSpecName "kube-api-access-gg6lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:23:58.983042 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:58.983006 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-util" (OuterVolumeSpecName: "util") pod "1951dbc6-f924-4a26-bcc8-78765468652b" (UID: "1951dbc6-f924-4a26-bcc8-78765468652b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:59.078383 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.078344 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg6lt\" (UniqueName: \"kubernetes.io/projected/1951dbc6-f924-4a26-bcc8-78765468652b-kube-api-access-gg6lt\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:59.078383 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.078374 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:59.078383 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.078389 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1951dbc6-f924-4a26-bcc8-78765468652b-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:59.719839 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.719810 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" Sep 08 16:23:59.720400 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.719842 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wg4bh" event={"ID":"be7c8429-b684-4d1f-9ed3-521a3acf11f4","Type":"ContainerDied","Data":"8f874585ddd1997b20e68d7f3bfff925d1970f005eb9a4f579bc1357952cc9a5"} Sep 08 16:23:59.720400 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.719928 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f874585ddd1997b20e68d7f3bfff925d1970f005eb9a4f579bc1357952cc9a5" Sep 08 16:23:59.721952 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.721920 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" event={"ID":"1951dbc6-f924-4a26-bcc8-78765468652b","Type":"ContainerDied","Data":"4d2c27fe3137c72df41aafd14ff05dd3a0b43b399aea7a76fc441595bac3245a"} Sep 08 16:23:59.722085 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.721951 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6svr7" Sep 08 16:23:59.722085 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.721981 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d2c27fe3137c72df41aafd14ff05dd3a0b43b399aea7a76fc441595bac3245a" Sep 08 16:23:59.850608 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.850581 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:23:59.886594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.886560 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdg97\" (UniqueName: \"kubernetes.io/projected/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-kube-api-access-vdg97\") pod \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " Sep 08 16:23:59.886743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.886680 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-bundle\") pod \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " Sep 08 16:23:59.886743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.886723 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-util\") pod \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\" (UID: \"32970005-f9e5-4fc5-a7c3-f75a7ddf786a\") " Sep 08 16:23:59.887128 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.887096 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-bundle" (OuterVolumeSpecName: "bundle") pod "32970005-f9e5-4fc5-a7c3-f75a7ddf786a" (UID: "32970005-f9e5-4fc5-a7c3-f75a7ddf786a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:59.888732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.888706 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-kube-api-access-vdg97" (OuterVolumeSpecName: "kube-api-access-vdg97") pod "32970005-f9e5-4fc5-a7c3-f75a7ddf786a" (UID: "32970005-f9e5-4fc5-a7c3-f75a7ddf786a"). InnerVolumeSpecName "kube-api-access-vdg97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:23:59.892323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.892298 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-util" (OuterVolumeSpecName: "util") pod "32970005-f9e5-4fc5-a7c3-f75a7ddf786a" (UID: "32970005-f9e5-4fc5-a7c3-f75a7ddf786a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:23:59.987976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.987884 2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-bundle\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:59.987976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.987916 2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-util\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:23:59.987976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:23:59.987925 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdg97\" (UniqueName: \"kubernetes.io/projected/32970005-f9e5-4fc5-a7c3-f75a7ddf786a-kube-api-access-vdg97\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:24:00.729146 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:24:00.729110 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" Sep 08 16:24:00.729146 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:24:00.729120 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67zdsh" event={"ID":"32970005-f9e5-4fc5-a7c3-f75a7ddf786a","Type":"ContainerDied","Data":"29a037275d832690d36bfca1abe07535b10acf0b8a51f22741d1f1250739d4a2"} Sep 08 16:24:00.729646 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:24:00.729175 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a037275d832690d36bfca1abe07535b10acf0b8a51f22741d1f1250739d4a2" Sep 08 16:25:22.168895 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.168847 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-twgcv"] Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169388 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerName="extract" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169408 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerName="extract" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169427 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1951dbc6-f924-4a26-bcc8-78765468652b" containerName="pull" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169437 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1951dbc6-f924-4a26-bcc8-78765468652b" containerName="pull" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169448 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerName="extract" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169458 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerName="extract" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169469 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1951dbc6-f924-4a26-bcc8-78765468652b" containerName="extract" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169477 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1951dbc6-f924-4a26-bcc8-78765468652b" containerName="extract" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169495 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerName="pull" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169503 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerName="pull" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169513 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerName="pull" Sep 08 16:25:22.169534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169521 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerName="pull" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169544 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a801e9c-1765-423a-acc4-798004ac1f93" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169553 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a801e9c-1765-423a-acc4-798004ac1f93" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169561 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a801e9c-1765-423a-acc4-798004ac1f93" containerName="extract" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169569 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a801e9c-1765-423a-acc4-798004ac1f93" containerName="extract" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169582 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169590 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169603 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1951dbc6-f924-4a26-bcc8-78765468652b" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169611 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1951dbc6-f924-4a26-bcc8-78765468652b" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169620 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169628 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerName="util" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169644 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a801e9c-1765-423a-acc4-798004ac1f93" containerName="pull" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169652 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a801e9c-1765-423a-acc4-798004ac1f93" containerName="pull" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169759 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a801e9c-1765-423a-acc4-798004ac1f93" containerName="extract" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169770 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7c8429-b684-4d1f-9ed3-521a3acf11f4" containerName="extract" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169780 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32970005-f9e5-4fc5-a7c3-f75a7ddf786a" containerName="extract" Sep 08 16:25:22.170124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.169789 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1951dbc6-f924-4a26-bcc8-78765468652b" containerName="extract" Sep 08 16:25:22.172018 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.171998 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" Sep 08 16:25:22.174585 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.174565 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 16:25:22.175382 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.175365 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 16:25:22.175484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.175385 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zwblx\"" Sep 08 16:25:22.179398 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.179312 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-twgcv"] Sep 08 16:25:22.296838 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.296801 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-cvlw7"] Sep 08 16:25:22.299878 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.299854 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cvlw7" Sep 08 16:25:22.307680 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.307655 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cvlw7"] Sep 08 16:25:22.322864 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.322826 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qss2g\" (UniqueName: \"kubernetes.io/projected/5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a-kube-api-access-qss2g\") pod \"authorino-7bcfd8f9cd-twgcv\" (UID: \"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" Sep 08 16:25:22.424269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.424145 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb47c\" (UniqueName: \"kubernetes.io/projected/4bed0646-4da1-4c7d-b91e-83c679cf07c6-kube-api-access-zb47c\") pod \"authorino-59c974bd48-cvlw7\" (UID: \"4bed0646-4da1-4c7d-b91e-83c679cf07c6\") " pod="kuadrant-system/authorino-59c974bd48-cvlw7" Sep 08 16:25:22.424269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.424191 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qss2g\" (UniqueName: \"kubernetes.io/projected/5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a-kube-api-access-qss2g\") pod \"authorino-7bcfd8f9cd-twgcv\" (UID: \"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" Sep 08 16:25:22.432529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.432492 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qss2g\" (UniqueName: \"kubernetes.io/projected/5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a-kube-api-access-qss2g\") pod \"authorino-7bcfd8f9cd-twgcv\" (UID: \"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" Sep 08 16:25:22.482415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.482375 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" Sep 08 16:25:22.525641 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.525604 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb47c\" (UniqueName: \"kubernetes.io/projected/4bed0646-4da1-4c7d-b91e-83c679cf07c6-kube-api-access-zb47c\") pod \"authorino-59c974bd48-cvlw7\" (UID: \"4bed0646-4da1-4c7d-b91e-83c679cf07c6\") " pod="kuadrant-system/authorino-59c974bd48-cvlw7" Sep 08 16:25:22.535192 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.535015 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb47c\" (UniqueName: \"kubernetes.io/projected/4bed0646-4da1-4c7d-b91e-83c679cf07c6-kube-api-access-zb47c\") pod \"authorino-59c974bd48-cvlw7\" (UID: \"4bed0646-4da1-4c7d-b91e-83c679cf07c6\") " pod="kuadrant-system/authorino-59c974bd48-cvlw7" Sep 08 16:25:22.609604 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.609569 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cvlw7" Sep 08 16:25:22.755421 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.755328 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-twgcv"] Sep 08 16:25:22.757855 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:25:22.757825 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ecad3c8_54aa_4c3d_8d50_15a8c09a7c2a.slice/crio-c46d17b652d6db2e56e22110e4a6e5916063936a972a0de600e1e7f55236ed95 WatchSource:0}: Error finding container c46d17b652d6db2e56e22110e4a6e5916063936a972a0de600e1e7f55236ed95: Status 404 returned error can't find the container with id c46d17b652d6db2e56e22110e4a6e5916063936a972a0de600e1e7f55236ed95 Sep 08 16:25:22.881157 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:22.881129 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cvlw7"] Sep 08 16:25:22.882863 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:25:22.882828 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bed0646_4da1_4c7d_b91e_83c679cf07c6.slice/crio-6b5f657f2668316b54999a3413a8394d7115eb2a04abcb60e7bdc1b4dba9ec63 WatchSource:0}: Error finding container 6b5f657f2668316b54999a3413a8394d7115eb2a04abcb60e7bdc1b4dba9ec63: Status 404 returned error can't find the container with id 6b5f657f2668316b54999a3413a8394d7115eb2a04abcb60e7bdc1b4dba9ec63 Sep 08 16:25:23.171233 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:23.171194 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cvlw7" event={"ID":"4bed0646-4da1-4c7d-b91e-83c679cf07c6","Type":"ContainerStarted","Data":"6b5f657f2668316b54999a3413a8394d7115eb2a04abcb60e7bdc1b4dba9ec63"} Sep 08 16:25:23.172426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:23.172396 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" event={"ID":"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a","Type":"ContainerStarted","Data":"c46d17b652d6db2e56e22110e4a6e5916063936a972a0de600e1e7f55236ed95"} Sep 08 16:25:26.194760 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:26.194721 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cvlw7" event={"ID":"4bed0646-4da1-4c7d-b91e-83c679cf07c6","Type":"ContainerStarted","Data":"1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f"} Sep 08 16:25:26.196295 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:26.196269 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" event={"ID":"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a","Type":"ContainerStarted","Data":"ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7"} Sep 08 16:25:26.212234 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:26.212173 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-cvlw7" podStartSLOduration=1.676155124 podStartE2EDuration="4.212157544s" totalImagesPullingTime="2.53600242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:26.209924927 +0000 UTC m=+759.804923686" watchObservedRunningTime="2026-09-08 16:25:26.212157544 +0000 UTC m=+759.807156328" Sep 08 16:25:26.225864 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:26.225812 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" podStartSLOduration=1.579346162 podStartE2EDuration="4.225791756s" totalImagesPullingTime="2.646445594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:26.222908053 +0000 UTC m=+759.817906816" watchObservedRunningTime="2026-09-08 16:25:26.225791756 +0000 UTC m=+759.820790515" Sep 08 16:25:26.253113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:26.253074 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-twgcv"] Sep 08 16:25:28.207024 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:28.206981 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" podUID="5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a" containerName="authorino" containerID="cri-o://ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7" gracePeriod=30 Sep 08 16:25:28.459135 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:28.459059 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" Sep 08 16:25:28.475609 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:28.475573 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qss2g\" (UniqueName: \"kubernetes.io/projected/5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a-kube-api-access-qss2g\") pod \"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a\" (UID: \"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a\") " Sep 08 16:25:28.477857 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:28.477818 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a-kube-api-access-qss2g" (OuterVolumeSpecName: "kube-api-access-qss2g") pod "5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a" (UID: "5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a"). InnerVolumeSpecName "kube-api-access-qss2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:28.576625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:28.576580 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qss2g\" (UniqueName: \"kubernetes.io/projected/5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a-kube-api-access-qss2g\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:25:29.213522 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.213427 2555 generic.go:356] "Generic (PLEG): container finished" podID="5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a" containerID="ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7" exitCode=0 Sep 08 16:25:29.213522 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.213469 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" Sep 08 16:25:29.214010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.213520 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" event={"ID":"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a","Type":"ContainerDied","Data":"ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7"} Sep 08 16:25:29.214010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.213579 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-twgcv" event={"ID":"5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a","Type":"ContainerDied","Data":"c46d17b652d6db2e56e22110e4a6e5916063936a972a0de600e1e7f55236ed95"} Sep 08 16:25:29.214010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.213603 2555 scope.go:117] "RemoveContainer" containerID="ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7" Sep 08 16:25:29.222499 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.222480 2555 scope.go:117] "RemoveContainer" containerID="ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7" Sep 08 16:25:29.222835 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:25:29.222811 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7\": container with ID starting with ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7 not found: ID does not exist" containerID="ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7" Sep 08 16:25:29.222882 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.222846 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7"} err="failed to get container status \"ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7\": rpc error: code = NotFound desc = could not find container \"ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7\": container with ID starting with ae24b6f02d8d006e1deaff9e4f7da644619815969ee061156da081fb6653c8a7 not found: ID does not exist" Sep 08 16:25:29.231576 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.231549 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-twgcv"] Sep 08 16:25:29.234636 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:29.234609 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-twgcv"] Sep 08 16:25:30.937000 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:30.936959 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a" path="/var/lib/kubelet/pods/5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a/volumes" Sep 08 16:25:52.081908 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.081872 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-k59b5"] Sep 08 16:25:52.082368 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.082230 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a" containerName="authorino" Sep 08 16:25:52.082368 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.082287 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a" containerName="authorino" Sep 08 16:25:52.082449 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.082379 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecad3c8-54aa-4c3d-8d50-15a8c09a7c2a" containerName="authorino" Sep 08 16:25:52.086849 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.086826 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-k59b5" Sep 08 16:25:52.097555 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.097524 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-k59b5"] Sep 08 16:25:52.190252 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.190215 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkd9\" (UniqueName: \"kubernetes.io/projected/d29a9aa3-3355-4e9a-881a-a855aed15ab6-kube-api-access-6rkd9\") pod \"authorino-557f876f69-k59b5\" (UID: \"d29a9aa3-3355-4e9a-881a-a855aed15ab6\") " pod="kuadrant-system/authorino-557f876f69-k59b5" Sep 08 16:25:52.222782 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.222745 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-k59b5"] Sep 08 16:25:52.223083 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:25:52.223059 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6rkd9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-k59b5" podUID="d29a9aa3-3355-4e9a-881a-a855aed15ab6" Sep 08 16:25:52.251538 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.251499 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nzwh6"] Sep 08 16:25:52.254832 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.254806 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:52.257518 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.257494 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 08 16:25:52.266145 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.266120 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nzwh6"] Sep 08 16:25:52.291320 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.291277 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkd9\" (UniqueName: \"kubernetes.io/projected/d29a9aa3-3355-4e9a-881a-a855aed15ab6-kube-api-access-6rkd9\") pod \"authorino-557f876f69-k59b5\" (UID: \"d29a9aa3-3355-4e9a-881a-a855aed15ab6\") " pod="kuadrant-system/authorino-557f876f69-k59b5" Sep 08 16:25:52.304475 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.304443 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkd9\" (UniqueName: \"kubernetes.io/projected/d29a9aa3-3355-4e9a-881a-a855aed15ab6-kube-api-access-6rkd9\") pod \"authorino-557f876f69-k59b5\" (UID: \"d29a9aa3-3355-4e9a-881a-a855aed15ab6\") " pod="kuadrant-system/authorino-557f876f69-k59b5" Sep 08 16:25:52.334207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.334123 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-k59b5" Sep 08 16:25:52.341775 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.341752 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-k59b5" Sep 08 16:25:52.345576 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.345548 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nzwh6"] Sep 08 16:25:52.345826 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:25:52.345808 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sbw5q tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" podUID="2a5dc0c4-ee4c-4397-bdde-69a3339310a8" Sep 08 16:25:52.391860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.391820 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbw5q\" (UniqueName: \"kubernetes.io/projected/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-kube-api-access-sbw5q\") pod \"authorino-6d4dfc649f-nzwh6\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:52.392053 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.391868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-tls-cert\") pod \"authorino-6d4dfc649f-nzwh6\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:52.492446 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.492403 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rkd9\" (UniqueName: \"kubernetes.io/projected/d29a9aa3-3355-4e9a-881a-a855aed15ab6-kube-api-access-6rkd9\") pod \"d29a9aa3-3355-4e9a-881a-a855aed15ab6\" (UID: \"d29a9aa3-3355-4e9a-881a-a855aed15ab6\") " Sep 08 16:25:52.492682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.492664 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbw5q\" (UniqueName: \"kubernetes.io/projected/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-kube-api-access-sbw5q\") pod \"authorino-6d4dfc649f-nzwh6\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:52.492776 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.492715 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-tls-cert\") pod \"authorino-6d4dfc649f-nzwh6\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:52.494574 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.494547 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29a9aa3-3355-4e9a-881a-a855aed15ab6-kube-api-access-6rkd9" (OuterVolumeSpecName: "kube-api-access-6rkd9") pod "d29a9aa3-3355-4e9a-881a-a855aed15ab6" (UID: "d29a9aa3-3355-4e9a-881a-a855aed15ab6"). InnerVolumeSpecName "kube-api-access-6rkd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:52.495009 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.494987 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-tls-cert\") pod \"authorino-6d4dfc649f-nzwh6\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:52.501593 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.501569 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbw5q\" (UniqueName: \"kubernetes.io/projected/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-kube-api-access-sbw5q\") pod \"authorino-6d4dfc649f-nzwh6\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:52.593479 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:52.593381 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkd9\" (UniqueName: \"kubernetes.io/projected/d29a9aa3-3355-4e9a-881a-a855aed15ab6-kube-api-access-6rkd9\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:25:53.340040 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.339944 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:53.340511 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.339944 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-k59b5" Sep 08 16:25:53.348201 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.348176 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:53.385671 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.385638 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-k59b5"] Sep 08 16:25:53.389322 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.389292 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-k59b5"] Sep 08 16:25:53.503080 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.503038 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-tls-cert\") pod \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " Sep 08 16:25:53.503080 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.503089 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbw5q\" (UniqueName: \"kubernetes.io/projected/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-kube-api-access-sbw5q\") pod \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\" (UID: \"2a5dc0c4-ee4c-4397-bdde-69a3339310a8\") " Sep 08 16:25:53.505189 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.505162 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2a5dc0c4-ee4c-4397-bdde-69a3339310a8" (UID: "2a5dc0c4-ee4c-4397-bdde-69a3339310a8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:25:53.505319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.505296 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-kube-api-access-sbw5q" (OuterVolumeSpecName: "kube-api-access-sbw5q") pod "2a5dc0c4-ee4c-4397-bdde-69a3339310a8" (UID: "2a5dc0c4-ee4c-4397-bdde-69a3339310a8"). InnerVolumeSpecName "kube-api-access-sbw5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:53.604025 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.603990 2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-tls-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:25:53.604025 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:53.604023 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbw5q\" (UniqueName: \"kubernetes.io/projected/2a5dc0c4-ee4c-4397-bdde-69a3339310a8-kube-api-access-sbw5q\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:25:54.345803 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:54.345768 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nzwh6" Sep 08 16:25:54.384311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:54.384275 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nzwh6"] Sep 08 16:25:54.387157 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:54.387128 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nzwh6"] Sep 08 16:25:54.936474 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:54.936442 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5dc0c4-ee4c-4397-bdde-69a3339310a8" path="/var/lib/kubelet/pods/2a5dc0c4-ee4c-4397-bdde-69a3339310a8/volumes" Sep 08 16:25:54.936710 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:54.936695 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29a9aa3-3355-4e9a-881a-a855aed15ab6" path="/var/lib/kubelet/pods/d29a9aa3-3355-4e9a-881a-a855aed15ab6/volumes" Sep 08 16:25:56.003107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.003072 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cvlw7"] Sep 08 16:25:56.003519 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.003274 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-cvlw7" podUID="4bed0646-4da1-4c7d-b91e-83c679cf07c6" containerName="authorino" containerID="cri-o://1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f" gracePeriod=30 Sep 08 16:25:56.239719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.239690 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cvlw7" Sep 08 16:25:56.327626 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.327527 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb47c\" (UniqueName: \"kubernetes.io/projected/4bed0646-4da1-4c7d-b91e-83c679cf07c6-kube-api-access-zb47c\") pod \"4bed0646-4da1-4c7d-b91e-83c679cf07c6\" (UID: \"4bed0646-4da1-4c7d-b91e-83c679cf07c6\") " Sep 08 16:25:56.329647 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.329616 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bed0646-4da1-4c7d-b91e-83c679cf07c6-kube-api-access-zb47c" (OuterVolumeSpecName: "kube-api-access-zb47c") pod "4bed0646-4da1-4c7d-b91e-83c679cf07c6" (UID: "4bed0646-4da1-4c7d-b91e-83c679cf07c6"). InnerVolumeSpecName "kube-api-access-zb47c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:25:56.358011 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.357974 2555 generic.go:356] "Generic (PLEG): container finished" podID="4bed0646-4da1-4c7d-b91e-83c679cf07c6" containerID="1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f" exitCode=0 Sep 08 16:25:56.358207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.358022 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cvlw7" Sep 08 16:25:56.358207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.358047 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cvlw7" event={"ID":"4bed0646-4da1-4c7d-b91e-83c679cf07c6","Type":"ContainerDied","Data":"1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f"} Sep 08 16:25:56.358207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.358078 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cvlw7" event={"ID":"4bed0646-4da1-4c7d-b91e-83c679cf07c6","Type":"ContainerDied","Data":"6b5f657f2668316b54999a3413a8394d7115eb2a04abcb60e7bdc1b4dba9ec63"} Sep 08 16:25:56.358207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.358094 2555 scope.go:117] "RemoveContainer" containerID="1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f" Sep 08 16:25:56.369498 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.368410 2555 scope.go:117] "RemoveContainer" containerID="1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f" Sep 08 16:25:56.370181 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:25:56.370152 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f\": container with ID starting with 1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f not found: ID does not exist" containerID="1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f" Sep 08 16:25:56.370384 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.370189 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f"} err="failed to get container status \"1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f\": rpc error: code = NotFound desc = could not find container \"1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f\": container with ID starting with 1b05e6f71e641558578d475b413a66de2f119970c4ab28fe179cdbfd5c3ae95f not found: ID does not exist" Sep 08 16:25:56.380509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.380477 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cvlw7"] Sep 08 16:25:56.385508 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.385481 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cvlw7"] Sep 08 16:25:56.428182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.428149 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb47c\" (UniqueName: \"kubernetes.io/projected/4bed0646-4da1-4c7d-b91e-83c679cf07c6-kube-api-access-zb47c\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:25:56.936848 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:25:56.936816 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bed0646-4da1-4c7d-b91e-83c679cf07c6" path="/var/lib/kubelet/pods/4bed0646-4da1-4c7d-b91e-83c679cf07c6/volumes" Sep 08 16:26:01.054281 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.054228 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587797bcc9-bw8rs"] Sep 08 16:26:01.054745 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.054643 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bed0646-4da1-4c7d-b91e-83c679cf07c6" containerName="authorino" Sep 08 16:26:01.054745 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.054654 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bed0646-4da1-4c7d-b91e-83c679cf07c6" containerName="authorino" Sep 08 16:26:01.054745 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.054715 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bed0646-4da1-4c7d-b91e-83c679cf07c6" containerName="authorino" Sep 08 16:26:01.059922 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.059893 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.063546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.063521 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdv66\"" Sep 08 16:26:01.063686 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.063564 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 16:26:01.064565 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.064547 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 16:26:01.064625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.064616 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 16:26:01.076419 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.076388 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587797bcc9-bw8rs"] Sep 08 16:26:01.177126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.177088 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-webhook-cert\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.177349 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.177137 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-metrics-certs\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.177349 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.177197 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzlhn\" (UniqueName: \"kubernetes.io/projected/0f252198-3fcd-4324-9cce-42145a51914b-kube-api-access-hzlhn\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.278707 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.278667 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-metrics-certs\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.278899 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.278718 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzlhn\" (UniqueName: \"kubernetes.io/projected/0f252198-3fcd-4324-9cce-42145a51914b-kube-api-access-hzlhn\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.278899 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.278839 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-webhook-cert\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.281142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.281115 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-metrics-certs\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.281142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.281142 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-webhook-cert\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.287682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.287652 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzlhn\" (UniqueName: \"kubernetes.io/projected/0f252198-3fcd-4324-9cce-42145a51914b-kube-api-access-hzlhn\") pod \"maas-controller-587797bcc9-bw8rs\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.371454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.371416 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:01.645541 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:01.645513 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587797bcc9-bw8rs"] Sep 08 16:26:01.648048 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:26:01.648021 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f252198_3fcd_4324_9cce_42145a51914b.slice/crio-33458e722a8944d5a4b0c44d4010e2040556fb3e30fc5f660941ea58f28590b6 WatchSource:0}: Error finding container 33458e722a8944d5a4b0c44d4010e2040556fb3e30fc5f660941ea58f28590b6: Status 404 returned error can't find the container with id 33458e722a8944d5a4b0c44d4010e2040556fb3e30fc5f660941ea58f28590b6 Sep 08 16:26:02.395182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:02.395137 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-bw8rs" event={"ID":"0f252198-3fcd-4324-9cce-42145a51914b","Type":"ContainerStarted","Data":"33458e722a8944d5a4b0c44d4010e2040556fb3e30fc5f660941ea58f28590b6"} Sep 08 16:26:04.413464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:04.413420 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-bw8rs" event={"ID":"0f252198-3fcd-4324-9cce-42145a51914b","Type":"ContainerStarted","Data":"bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876"} Sep 08 16:26:04.413945 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:04.413520 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:04.433657 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:04.433600 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-587797bcc9-bw8rs" podStartSLOduration=1.553062541 podStartE2EDuration="3.433581362s" totalImagesPullingTime="1.880518821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:04.431818556 +0000 UTC m=+798.026817336" watchObservedRunningTime="2026-09-08 16:26:04.433581362 +0000 UTC m=+798.028580122" Sep 08 16:26:15.422202 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:15.422117 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:26:37.334277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.334231 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg"] Sep 08 16:26:37.340548 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.340523 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.356091 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.356066 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 16:26:37.360799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.360778 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 16:26:37.361089 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.361072 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 16:26:37.361330 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.361316 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksssl\"" Sep 08 16:26:37.371910 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.371888 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 16:26:37.394083 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.394048 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgwp\" (UniqueName: \"kubernetes.io/projected/a405e5a9-1c8a-4d55-a926-5c09a7668c24-kube-api-access-6hgwp\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.394195 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.394130 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-metrics-certs\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.394195 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.394154 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a405e5a9-1c8a-4d55-a926-5c09a7668c24-openshift-service-ca\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.394195 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.394172 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-maas-api-tls\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.431401 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.431360 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg"] Sep 08 16:26:37.494810 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.494769 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-metrics-certs\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.494810 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.494815 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a405e5a9-1c8a-4d55-a926-5c09a7668c24-openshift-service-ca\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.495087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.494837 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-maas-api-tls\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.495087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.494875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgwp\" (UniqueName: \"kubernetes.io/projected/a405e5a9-1c8a-4d55-a926-5c09a7668c24-kube-api-access-6hgwp\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.495514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.495484 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a405e5a9-1c8a-4d55-a926-5c09a7668c24-openshift-service-ca\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.497391 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.497361 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-metrics-certs\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.497649 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.497630 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-maas-api-tls\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.511233 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.511201 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgwp\" (UniqueName: \"kubernetes.io/projected/a405e5a9-1c8a-4d55-a926-5c09a7668c24-kube-api-access-6hgwp\") pod \"maas-api-5df4497bcd-45qkg\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.650416 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.650368 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:37.933921 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:26:37.933893 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda405e5a9_1c8a_4d55_a926_5c09a7668c24.slice/crio-b43203250ff1305c1cec3c61649888726a1db779ab101927fcc43ed1f7566ee1 WatchSource:0}: Error finding container b43203250ff1305c1cec3c61649888726a1db779ab101927fcc43ed1f7566ee1: Status 404 returned error can't find the container with id b43203250ff1305c1cec3c61649888726a1db779ab101927fcc43ed1f7566ee1 Sep 08 16:26:37.934560 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:37.934539 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg"] Sep 08 16:26:38.599082 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:38.599045 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" event={"ID":"a405e5a9-1c8a-4d55-a926-5c09a7668c24","Type":"ContainerStarted","Data":"b43203250ff1305c1cec3c61649888726a1db779ab101927fcc43ed1f7566ee1"} Sep 08 16:26:40.612940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:40.612901 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" event={"ID":"a405e5a9-1c8a-4d55-a926-5c09a7668c24","Type":"ContainerStarted","Data":"7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca"} Sep 08 16:26:40.613358 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:40.612984 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:26:40.636345 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:40.636279 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" podStartSLOduration=1.54865948 podStartE2EDuration="3.636236852s" totalImagesPullingTime="2.087577372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:40.635279795 +0000 UTC m=+834.230278575" watchObservedRunningTime="2026-09-08 16:26:40.636236852 +0000 UTC m=+834.231235613" Sep 08 16:26:46.624217 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:26:46.624183 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:27:06.616649 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.616611 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82"] Sep 08 16:27:06.621190 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.621165 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.624859 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.624834 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 16:27:06.624993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.624888 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 08 16:27:06.626230 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.626212 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 16:27:06.626427 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.626231 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xc4pf\"" Sep 08 16:27:06.642087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.642053 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82"] Sep 08 16:27:06.659978 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.659943 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.660179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.659984 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95f592-cea0-42bd-9f62-42baf20cad23-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.660179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.660045 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.660179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.660065 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.660179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.660152 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.660426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.660190 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9xj\" (UniqueName: \"kubernetes.io/projected/ea95f592-cea0-42bd-9f62-42baf20cad23-kube-api-access-rz9xj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.760880 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.760844 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.760880 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.760880 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9xj\" (UniqueName: \"kubernetes.io/projected/ea95f592-cea0-42bd-9f62-42baf20cad23-kube-api-access-rz9xj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.761106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.761004 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.761106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.761044 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95f592-cea0-42bd-9f62-42baf20cad23-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.761106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.761083 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.761278 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.761107 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.761437 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.761411 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.761504 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.761465 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.761549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.761533 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.763092 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.763068 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea95f592-cea0-42bd-9f62-42baf20cad23-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.763445 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.763428 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea95f592-cea0-42bd-9f62-42baf20cad23-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.772886 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.772863 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9xj\" (UniqueName: \"kubernetes.io/projected/ea95f592-cea0-42bd-9f62-42baf20cad23-kube-api-access-rz9xj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hq82\" (UID: \"ea95f592-cea0-42bd-9f62-42baf20cad23\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:06.932180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:06.932072 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:07.222717 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:07.222689 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82"] Sep 08 16:27:07.224770 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:27:07.224733 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea95f592_cea0_42bd_9f62_42baf20cad23.slice/crio-879687b0fdc9fa5b8930adde00be1a6feda992f536369a8be9294ca6323a13d4 WatchSource:0}: Error finding container 879687b0fdc9fa5b8930adde00be1a6feda992f536369a8be9294ca6323a13d4: Status 404 returned error can't find the container with id 879687b0fdc9fa5b8930adde00be1a6feda992f536369a8be9294ca6323a13d4 Sep 08 16:27:07.763054 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:07.763014 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" event={"ID":"ea95f592-cea0-42bd-9f62-42baf20cad23","Type":"ContainerStarted","Data":"879687b0fdc9fa5b8930adde00be1a6feda992f536369a8be9294ca6323a13d4"} Sep 08 16:27:08.770857 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:08.770812 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" event={"ID":"ea95f592-cea0-42bd-9f62-42baf20cad23","Type":"ContainerStarted","Data":"8e8422026a2e46b07ead75022fa32702ac0d11d47e0b9392e9a92cd2ad98d2dd"} Sep 08 16:27:08.795530 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:08.795464 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" podStartSLOduration=1.8068359649999999 podStartE2EDuration="2.79544744s" totalImagesPullingTime="988.611475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:08.792580789 +0000 UTC m=+862.387579563" watchObservedRunningTime="2026-09-08 16:27:08.79544744 +0000 UTC m=+862.390446199" Sep 08 16:27:16.937302 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:16.937238 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:16.937780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:16.937312 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:16.946286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:16.946253 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:17.831798 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:17.831762 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hq82" Sep 08 16:27:21.933181 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:21.933148 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp"] Sep 08 16:27:21.935843 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:21.935825 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:21.942620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:21.942566 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 08 16:27:22.001804 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.001769 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.001987 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.001824 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.002033 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.002014 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr56t\" (UniqueName: \"kubernetes.io/projected/6113ccbe-9a15-4d94-a476-45fe03b49f97-kube-api-access-sr56t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.002074 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.002067 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6113ccbe-9a15-4d94-a476-45fe03b49f97-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.002182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.002162 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.002234 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.002222 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.068955 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.068915 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp"] Sep 08 16:27:22.103601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.103560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.103601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.103604 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.103860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.103628 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.103860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.103658 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr56t\" (UniqueName: \"kubernetes.io/projected/6113ccbe-9a15-4d94-a476-45fe03b49f97-kube-api-access-sr56t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.103860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.103709 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6113ccbe-9a15-4d94-a476-45fe03b49f97-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.103860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.103768 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.104132 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.104047 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.104132 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.104072 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.104234 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.104173 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.106158 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.106135 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6113ccbe-9a15-4d94-a476-45fe03b49f97-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.106324 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.106308 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6113ccbe-9a15-4d94-a476-45fe03b49f97-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.144685 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.144642 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr56t\" (UniqueName: \"kubernetes.io/projected/6113ccbe-9a15-4d94-a476-45fe03b49f97-kube-api-access-sr56t\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4jsmp\" (UID: \"6113ccbe-9a15-4d94-a476-45fe03b49f97\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.246724 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.246630 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:22.537902 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.537876 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp"] Sep 08 16:27:22.540235 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:27:22.540198 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6113ccbe_9a15_4d94_a476_45fe03b49f97.slice/crio-63755e1e41364c5481ef7fcf72297ed096352c5e1d638fe9d69f0ba165c860fe WatchSource:0}: Error finding container 63755e1e41364c5481ef7fcf72297ed096352c5e1d638fe9d69f0ba165c860fe: Status 404 returned error can't find the container with id 63755e1e41364c5481ef7fcf72297ed096352c5e1d638fe9d69f0ba165c860fe Sep 08 16:27:22.848212 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.848179 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" event={"ID":"6113ccbe-9a15-4d94-a476-45fe03b49f97","Type":"ContainerStarted","Data":"fa50709d437ae74cf0533d04489f43fd483c15857177e904b433636cccd0fd87"} Sep 08 16:27:22.848382 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.848232 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" event={"ID":"6113ccbe-9a15-4d94-a476-45fe03b49f97","Type":"ContainerStarted","Data":"63755e1e41364c5481ef7fcf72297ed096352c5e1d638fe9d69f0ba165c860fe"} Sep 08 16:27:22.877976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:22.877918 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" podStartSLOduration=1.6330364990000001 podStartE2EDuration="1.877899914s" totalImagesPullingTime="244.863415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:22.876663425 +0000 UTC m=+876.471662204" watchObservedRunningTime="2026-09-08 16:27:22.877899914 +0000 UTC m=+876.472898672" Sep 08 16:27:32.247540 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:32.247497 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:32.248060 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:32.247568 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:32.263808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:32.260806 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:27:32.919929 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:27:32.919896 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4jsmp" Sep 08 16:29:05.491077 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.491037 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84b96d9cf5-dbzbm"] Sep 08 16:29:05.494740 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.494720 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:05.497365 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.497337 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 16:29:05.497505 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.497366 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 08 16:29:05.498466 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.498449 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zwblx\"" Sep 08 16:29:05.498524 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.498499 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 16:29:05.508180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.508150 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84b96d9cf5-dbzbm"] Sep 08 16:29:05.634539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.634497 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6af6caa1-8971-4149-8823-dfb686d1c507-tls-cert\") pod \"authorino-84b96d9cf5-dbzbm\" (UID: \"6af6caa1-8971-4149-8823-dfb686d1c507\") " pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:05.634723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.634573 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt9dv\" (UniqueName: \"kubernetes.io/projected/6af6caa1-8971-4149-8823-dfb686d1c507-kube-api-access-bt9dv\") pod \"authorino-84b96d9cf5-dbzbm\" (UID: \"6af6caa1-8971-4149-8823-dfb686d1c507\") " pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:05.735549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.735513 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt9dv\" (UniqueName: \"kubernetes.io/projected/6af6caa1-8971-4149-8823-dfb686d1c507-kube-api-access-bt9dv\") pod \"authorino-84b96d9cf5-dbzbm\" (UID: \"6af6caa1-8971-4149-8823-dfb686d1c507\") " pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:05.735722 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.735591 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6af6caa1-8971-4149-8823-dfb686d1c507-tls-cert\") pod \"authorino-84b96d9cf5-dbzbm\" (UID: \"6af6caa1-8971-4149-8823-dfb686d1c507\") " pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:05.738063 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.738033 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6af6caa1-8971-4149-8823-dfb686d1c507-tls-cert\") pod \"authorino-84b96d9cf5-dbzbm\" (UID: \"6af6caa1-8971-4149-8823-dfb686d1c507\") " pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:05.744963 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.744892 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt9dv\" (UniqueName: \"kubernetes.io/projected/6af6caa1-8971-4149-8823-dfb686d1c507-kube-api-access-bt9dv\") pod \"authorino-84b96d9cf5-dbzbm\" (UID: \"6af6caa1-8971-4149-8823-dfb686d1c507\") " pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:05.804847 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:05.804807 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" Sep 08 16:29:06.079475 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:06.079447 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84b96d9cf5-dbzbm"] Sep 08 16:29:06.080959 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:29:06.080929 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6af6caa1_8971_4149_8823_dfb686d1c507.slice/crio-b68eb4430e12c8f04512ea1470601feb75eb2961b286f34a4b13471dd5c503ef WatchSource:0}: Error finding container b68eb4430e12c8f04512ea1470601feb75eb2961b286f34a4b13471dd5c503ef: Status 404 returned error can't find the container with id b68eb4430e12c8f04512ea1470601feb75eb2961b286f34a4b13471dd5c503ef Sep 08 16:29:06.082677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:06.082659 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:29:06.434889 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:06.434853 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" event={"ID":"6af6caa1-8971-4149-8823-dfb686d1c507","Type":"ContainerStarted","Data":"b68eb4430e12c8f04512ea1470601feb75eb2961b286f34a4b13471dd5c503ef"} Sep 08 16:29:07.451733 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:07.451690 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" event={"ID":"6af6caa1-8971-4149-8823-dfb686d1c507","Type":"ContainerStarted","Data":"5589392ae92d06443235992466275e784621b2bd697d3f480e6a3c9c32fa9d16"} Sep 08 16:29:07.491827 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:07.491768 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84b96d9cf5-dbzbm" podStartSLOduration=1.616417091 podStartE2EDuration="2.491750699s" totalImagesPullingTime="875.333608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:07.490496716 +0000 UTC m=+981.085495478" watchObservedRunningTime="2026-09-08 16:29:07.491750699 +0000 UTC m=+981.086749458" Sep 08 16:29:27.931275 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:27.931182 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:27.934708 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:27.934688 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:27.939327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:27.939305 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wdbqb\"" Sep 08 16:29:27.949800 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:27.949769 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:28.036919 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:28.036876 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4972\" (UniqueName: \"kubernetes.io/projected/833a0511-4883-4f79-9304-9828c8308523-kube-api-access-k4972\") pod \"test-curl-7911-140341053901872\" (UID: \"833a0511-4883-4f79-9304-9828c8308523\") " pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:28.137704 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:28.137659 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4972\" (UniqueName: \"kubernetes.io/projected/833a0511-4883-4f79-9304-9828c8308523-kube-api-access-k4972\") pod \"test-curl-7911-140341053901872\" (UID: \"833a0511-4883-4f79-9304-9828c8308523\") " pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:28.148856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:28.148829 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4972\" (UniqueName: \"kubernetes.io/projected/833a0511-4883-4f79-9304-9828c8308523-kube-api-access-k4972\") pod \"test-curl-7911-140341053901872\" (UID: \"833a0511-4883-4f79-9304-9828c8308523\") " pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:28.244102 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:28.244007 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:28.522638 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:28.522602 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:28.523292 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:29:28.523262 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod833a0511_4883_4f79_9304_9828c8308523.slice/crio-8603d8e0d310791464466ed06ae7c98f045f9d2ae1696c70b5c8f5b3f3bd4efb WatchSource:0}: Error finding container 8603d8e0d310791464466ed06ae7c98f045f9d2ae1696c70b5c8f5b3f3bd4efb: Status 404 returned error can't find the container with id 8603d8e0d310791464466ed06ae7c98f045f9d2ae1696c70b5c8f5b3f3bd4efb Sep 08 16:29:28.569117 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:28.569079 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7911-140341053901872" event={"ID":"833a0511-4883-4f79-9304-9828c8308523","Type":"ContainerStarted","Data":"8603d8e0d310791464466ed06ae7c98f045f9d2ae1696c70b5c8f5b3f3bd4efb"} Sep 08 16:29:29.576759 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:29.576670 2555 generic.go:356] "Generic (PLEG): container finished" podID="833a0511-4883-4f79-9304-9828c8308523" containerID="0e08ed39ceb8d600f08f9fd4bd70ee9eb6254915de227586efe8b903a44a0777" exitCode=0 Sep 08 16:29:29.577125 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:29.576755 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7911-140341053901872" event={"ID":"833a0511-4883-4f79-9304-9828c8308523","Type":"ContainerDied","Data":"0e08ed39ceb8d600f08f9fd4bd70ee9eb6254915de227586efe8b903a44a0777"} Sep 08 16:29:30.718870 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.718841 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:30.788874 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.788839 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7911-140341053901872_833a0511-4883-4f79-9304-9828c8308523/test-curl-7911-140341053901872/0.log" Sep 08 16:29:30.822499 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.822460 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:30.836918 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.836829 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:30.864902 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.864856 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4972\" (UniqueName: \"kubernetes.io/projected/833a0511-4883-4f79-9304-9828c8308523-kube-api-access-k4972\") pod \"833a0511-4883-4f79-9304-9828c8308523\" (UID: \"833a0511-4883-4f79-9304-9828c8308523\") " Sep 08 16:29:30.867070 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.867038 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833a0511-4883-4f79-9304-9828c8308523-kube-api-access-k4972" (OuterVolumeSpecName: "kube-api-access-k4972") pod "833a0511-4883-4f79-9304-9828c8308523" (UID: "833a0511-4883-4f79-9304-9828c8308523"). InnerVolumeSpecName "kube-api-access-k4972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:29:30.943399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.940187 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833a0511-4883-4f79-9304-9828c8308523" path="/var/lib/kubelet/pods/833a0511-4883-4f79-9304-9828c8308523/volumes" Sep 08 16:29:30.966604 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.966563 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4972\" (UniqueName: \"kubernetes.io/projected/833a0511-4883-4f79-9304-9828c8308523-kube-api-access-k4972\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:29:30.978842 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.978782 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:30.979485 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.979464 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833a0511-4883-4f79-9304-9828c8308523" containerName="test-curl-7911-140341053901872" Sep 08 16:29:30.979534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.979489 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="833a0511-4883-4f79-9304-9828c8308523" containerName="test-curl-7911-140341053901872" Sep 08 16:29:30.979627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.979615 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="833a0511-4883-4f79-9304-9828c8308523" containerName="test-curl-7911-140341053901872" Sep 08 16:29:30.983375 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.983348 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:30.998654 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:30.998620 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:31.168232 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:31.168185 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkvb\" (UniqueName: \"kubernetes.io/projected/930c8f45-d4aa-403b-9a4f-d787eba8bec2-kube-api-access-cdkvb\") pod \"test-curl-7911-140341053901872\" (UID: \"930c8f45-d4aa-403b-9a4f-d787eba8bec2\") " pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:31.269698 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:31.269655 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkvb\" (UniqueName: \"kubernetes.io/projected/930c8f45-d4aa-403b-9a4f-d787eba8bec2-kube-api-access-cdkvb\") pod \"test-curl-7911-140341053901872\" (UID: \"930c8f45-d4aa-403b-9a4f-d787eba8bec2\") " pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:31.285770 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:31.285739 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkvb\" (UniqueName: \"kubernetes.io/projected/930c8f45-d4aa-403b-9a4f-d787eba8bec2-kube-api-access-cdkvb\") pod \"test-curl-7911-140341053901872\" (UID: \"930c8f45-d4aa-403b-9a4f-d787eba8bec2\") " pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:31.293648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:31.293621 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:31.583426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:31.583387 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:31.584762 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:29:31.584736 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod930c8f45_d4aa_403b_9a4f_d787eba8bec2.slice/crio-4d13ecf4a529cc68b99204bbc904d4eb86ea94c52baa9672d514dc9e1536af8d WatchSource:0}: Error finding container 4d13ecf4a529cc68b99204bbc904d4eb86ea94c52baa9672d514dc9e1536af8d: Status 404 returned error can't find the container with id 4d13ecf4a529cc68b99204bbc904d4eb86ea94c52baa9672d514dc9e1536af8d Sep 08 16:29:31.589638 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:31.589620 2555 scope.go:117] "RemoveContainer" containerID="0e08ed39ceb8d600f08f9fd4bd70ee9eb6254915de227586efe8b903a44a0777" Sep 08 16:29:31.589733 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:31.589624 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:32.598432 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:32.598391 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7911-140341053901872" event={"ID":"930c8f45-d4aa-403b-9a4f-d787eba8bec2","Type":"ContainerStarted","Data":"9733e89b28788ed4445e3343b33d3d8846f964270f692aa1a0ebc44130cc9803"} Sep 08 16:29:32.598880 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:32.598444 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7911-140341053901872" event={"ID":"930c8f45-d4aa-403b-9a4f-d787eba8bec2","Type":"ContainerStarted","Data":"4d13ecf4a529cc68b99204bbc904d4eb86ea94c52baa9672d514dc9e1536af8d"} Sep 08 16:29:32.627614 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:32.627562 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7911-140341053901872" podStartSLOduration=1.729378962 podStartE2EDuration="2.627544346s" totalImagesPullingTime="898.165384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:32.626433739 +0000 UTC m=+1006.221432498" watchObservedRunningTime="2026-09-08 16:29:32.627544346 +0000 UTC m=+1006.222543105" Sep 08 16:29:32.731047 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:32.731009 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7911-140341053901872_930c8f45-d4aa-403b-9a4f-d787eba8bec2/test-curl-7911-140341053901872/0.log" Sep 08 16:29:33.605175 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:33.605134 2555 generic.go:356] "Generic (PLEG): container finished" podID="930c8f45-d4aa-403b-9a4f-d787eba8bec2" containerID="9733e89b28788ed4445e3343b33d3d8846f964270f692aa1a0ebc44130cc9803" exitCode=0 Sep 08 16:29:33.605705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:33.605283 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7911-140341053901872" event={"ID":"930c8f45-d4aa-403b-9a4f-d787eba8bec2","Type":"ContainerDied","Data":"9733e89b28788ed4445e3343b33d3d8846f964270f692aa1a0ebc44130cc9803"} Sep 08 16:29:34.743100 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:34.743072 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:34.781217 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:34.781179 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:34.785423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:34.785388 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7911-140341053901872"] Sep 08 16:29:34.799090 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:34.799057 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdkvb\" (UniqueName: \"kubernetes.io/projected/930c8f45-d4aa-403b-9a4f-d787eba8bec2-kube-api-access-cdkvb\") pod \"930c8f45-d4aa-403b-9a4f-d787eba8bec2\" (UID: \"930c8f45-d4aa-403b-9a4f-d787eba8bec2\") " Sep 08 16:29:34.801136 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:34.801100 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930c8f45-d4aa-403b-9a4f-d787eba8bec2-kube-api-access-cdkvb" (OuterVolumeSpecName: "kube-api-access-cdkvb") pod "930c8f45-d4aa-403b-9a4f-d787eba8bec2" (UID: "930c8f45-d4aa-403b-9a4f-d787eba8bec2"). InnerVolumeSpecName "kube-api-access-cdkvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:29:34.900104 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:34.900011 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkvb\" (UniqueName: \"kubernetes.io/projected/930c8f45-d4aa-403b-9a4f-d787eba8bec2-kube-api-access-cdkvb\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:29:34.937415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:34.937381 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930c8f45-d4aa-403b-9a4f-d787eba8bec2" path="/var/lib/kubelet/pods/930c8f45-d4aa-403b-9a4f-d787eba8bec2/volumes" Sep 08 16:29:35.606730 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.606691 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b"] Sep 08 16:29:35.607332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.607314 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930c8f45-d4aa-403b-9a4f-d787eba8bec2" containerName="test-curl-7911-140341053901872" Sep 08 16:29:35.607385 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.607333 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="930c8f45-d4aa-403b-9a4f-d787eba8bec2" containerName="test-curl-7911-140341053901872" Sep 08 16:29:35.607428 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.607413 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="930c8f45-d4aa-403b-9a4f-d787eba8bec2" containerName="test-curl-7911-140341053901872" Sep 08 16:29:35.610625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.610607 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.613339 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.613316 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-293166-openshift-default-dockercfg-xbmd8\"" Sep 08 16:29:35.619041 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.619021 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7911-140341053901872" Sep 08 16:29:35.619167 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.619071 2555 scope.go:117] "RemoveContainer" containerID="9733e89b28788ed4445e3343b33d3d8846f964270f692aa1a0ebc44130cc9803" Sep 08 16:29:35.630076 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.630040 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b"] Sep 08 16:29:35.708541 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708492 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-socket\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.708541 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708545 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdg27\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-kube-api-access-cdg27\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.708826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708601 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-token\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.708826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708654 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-credential-socket\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.708826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708724 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-certs\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.708826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708774 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-data\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.708826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708814 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-envoy\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.709115 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708837 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4076516-c2ce-4d4c-b6de-78c94c179e41-istiod-ca-cert\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.709115 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.708878 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-podinfo\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.809535 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809489 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-podinfo\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809552 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-socket\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809603 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdg27\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-kube-api-access-cdg27\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809638 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-token\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809685 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-credential-socket\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809727 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-certs\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809776 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-data\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809802 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-envoy\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.809826 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4076516-c2ce-4d4c-b6de-78c94c179e41-istiod-ca-cert\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810467 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.810107 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-socket\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810467 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.810141 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-certs\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810467 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.810222 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-data\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810467 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.810432 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-credential-socket\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.810629 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.810605 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4076516-c2ce-4d4c-b6de-78c94c179e41-istiod-ca-cert\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.812057 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.812033 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-envoy\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.812338 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.812317 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-podinfo\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.829022 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.828974 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-token\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.829178 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.829023 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdg27\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-kube-api-access-cdg27\") pod \"e2e-autores-293166-openshift-default-599d8647fb-gp64b\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:35.921906 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:35.921804 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:36.220514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.220483 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b"] Sep 08 16:29:36.223180 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:29:36.223153 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4076516_c2ce_4d4c_b6de_78c94c179e41.slice/crio-33b47b140ff707f713c65b1d2d30be0b7a671bebd548b59fa2443f61a4891b35 WatchSource:0}: Error finding container 33b47b140ff707f713c65b1d2d30be0b7a671bebd548b59fa2443f61a4891b35: Status 404 returned error can't find the container with id 33b47b140ff707f713c65b1d2d30be0b7a671bebd548b59fa2443f61a4891b35 Sep 08 16:29:36.226111 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.226071 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:29:36.226214 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.226147 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:29:36.226214 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.226176 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:29:36.628539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.628498 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" event={"ID":"b4076516-c2ce-4d4c-b6de-78c94c179e41","Type":"ContainerStarted","Data":"405fda300f9af6898e867a779323810cdf4fa71d14479f6ea8f4179375e091f8"} Sep 08 16:29:36.628721 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.628562 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" event={"ID":"b4076516-c2ce-4d4c-b6de-78c94c179e41","Type":"ContainerStarted","Data":"33b47b140ff707f713c65b1d2d30be0b7a671bebd548b59fa2443f61a4891b35"} Sep 08 16:29:36.666670 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.666588 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podStartSLOduration=1.666564881 podStartE2EDuration="1.666564881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:36.664054046 +0000 UTC m=+1010.259052822" watchObservedRunningTime="2026-09-08 16:29:36.666564881 +0000 UTC m=+1010.261563643" Sep 08 16:29:36.922587 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:36.922500 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:37.927403 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:37.927375 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:38.640137 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.640101 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:38.641004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.640982 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:38.840155 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.840119 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn"] Sep 08 16:29:38.843612 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.843590 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:38.847802 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.847776 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ec338109-serving-cert\"" Sep 08 16:29:38.867567 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.867533 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn"] Sep 08 16:29:38.941510 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.941424 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-metrics-certs\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:38.941510 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.941479 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhd4\" (UniqueName: \"kubernetes.io/projected/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-kube-api-access-9hhd4\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:38.941911 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.941553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-maas-api-tls\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:38.941911 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:38.941571 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-openshift-service-ca\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.042357 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.042320 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-openshift-service-ca\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.042570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.042377 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-metrics-certs\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.042570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.042425 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhd4\" (UniqueName: \"kubernetes.io/projected/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-kube-api-access-9hhd4\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.042570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.042514 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-maas-api-tls\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.043059 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.043035 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-openshift-service-ca\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.044951 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.044923 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-metrics-certs\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.045111 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.045087 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-maas-api-tls\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.058052 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.058023 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhd4\" (UniqueName: \"kubernetes.io/projected/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-kube-api-access-9hhd4\") pod \"maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.154640 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.154596 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:39.458082 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.458051 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn"] Sep 08 16:29:39.460266 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:29:39.460226 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d2d3f0_36be_47a9_9d9a_abd2bf92f3cd.slice/crio-1e83da9045d114bd5f8e59041b8a52dffc315ad4e33034b3395ed06adc5298b1 WatchSource:0}: Error finding container 1e83da9045d114bd5f8e59041b8a52dffc315ad4e33034b3395ed06adc5298b1: Status 404 returned error can't find the container with id 1e83da9045d114bd5f8e59041b8a52dffc315ad4e33034b3395ed06adc5298b1 Sep 08 16:29:39.647884 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:39.647848 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" event={"ID":"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd","Type":"ContainerStarted","Data":"1e83da9045d114bd5f8e59041b8a52dffc315ad4e33034b3395ed06adc5298b1"} Sep 08 16:29:40.655117 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.655078 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" event={"ID":"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd","Type":"ContainerStarted","Data":"541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb"} Sep 08 16:29:40.655605 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.655397 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:40.693213 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.693152 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" podStartSLOduration=2.353449862 podStartE2EDuration="2.69313573s" totalImagesPullingTime="339.685868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:40.690939991 +0000 UTC m=+1014.285938763" watchObservedRunningTime="2026-09-08 16:29:40.69313573 +0000 UTC m=+1014.288134485" Sep 08 16:29:40.798318 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.798272 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b"] Sep 08 16:29:40.801942 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.801920 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.809942 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.809919 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-293166-serving-cert\"" Sep 08 16:29:40.820893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.820863 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b"] Sep 08 16:29:40.858364 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.858320 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-openshift-service-ca\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.858568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.858375 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-metrics-certs\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.858568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.858408 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k792z\" (UniqueName: \"kubernetes.io/projected/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-kube-api-access-k792z\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.858568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.858459 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.959305 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.959184 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.959476 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.959372 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-openshift-service-ca\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.959476 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:29:40.959382 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-293166-serving-cert: secret "maas-api-e2e-autores-293166-serving-cert" not found Sep 08 16:29:40.959476 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.959404 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-metrics-certs\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.959476 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.959448 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k792z\" (UniqueName: \"kubernetes.io/projected/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-kube-api-access-k792z\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.959683 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:29:40.959508 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls podName:1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0 nodeName:}" failed. No retries permitted until 2026-09-08 16:29:41.459479359 +0000 UTC m=+1015.054478096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls") pod "maas-api-e2e-autores-293166-85c4865bdf-jth2b" (UID: "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0") : secret "maas-api-e2e-autores-293166-serving-cert" not found Sep 08 16:29:40.959974 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.959954 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-openshift-service-ca\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.961911 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.961886 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-metrics-certs\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:40.987576 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:40.987543 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k792z\" (UniqueName: \"kubernetes.io/projected/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-kube-api-access-k792z\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:41.464642 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:41.464601 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:41.467054 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:41.467030 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls\") pod \"maas-api-e2e-autores-293166-85c4865bdf-jth2b\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:41.713612 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:41.713568 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:42.006647 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:42.006621 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b"] Sep 08 16:29:42.008679 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:29:42.008655 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5c52d1_ee4f_404f_b88e_b5515ebcbfd0.slice/crio-94369d53d97440ccef7ac6be874cf02d72e8f8b8e191a6f41ee1b87dc3a1d10b WatchSource:0}: Error finding container 94369d53d97440ccef7ac6be874cf02d72e8f8b8e191a6f41ee1b87dc3a1d10b: Status 404 returned error can't find the container with id 94369d53d97440ccef7ac6be874cf02d72e8f8b8e191a6f41ee1b87dc3a1d10b Sep 08 16:29:42.668906 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:42.668806 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" event={"ID":"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0","Type":"ContainerStarted","Data":"74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60"} Sep 08 16:29:42.668906 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:42.668871 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" event={"ID":"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0","Type":"ContainerStarted","Data":"94369d53d97440ccef7ac6be874cf02d72e8f8b8e191a6f41ee1b87dc3a1d10b"} Sep 08 16:29:42.669118 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:42.668913 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:42.697707 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:42.697649 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" podStartSLOduration=2.35001367 podStartE2EDuration="2.69763242s" totalImagesPullingTime="347.61875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:42.6954166 +0000 UTC m=+1016.290415385" watchObservedRunningTime="2026-09-08 16:29:42.69763242 +0000 UTC m=+1016.292631179" Sep 08 16:29:46.667311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:46.667280 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:29:48.680732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:48.680698 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:29:58.763065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:58.763026 2555 generic.go:356] "Generic (PLEG): container finished" podID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerID="405fda300f9af6898e867a779323810cdf4fa71d14479f6ea8f4179375e091f8" exitCode=0 Sep 08 16:29:58.763569 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:58.763101 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" event={"ID":"b4076516-c2ce-4d4c-b6de-78c94c179e41","Type":"ContainerDied","Data":"405fda300f9af6898e867a779323810cdf4fa71d14479f6ea8f4179375e091f8"} Sep 08 16:29:58.763620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:58.763606 2555 scope.go:117] "RemoveContainer" containerID="405fda300f9af6898e867a779323810cdf4fa71d14479f6ea8f4179375e091f8" Sep 08 16:29:58.764671 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:58.764641 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:29:58.764730 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:58.764717 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:29:58.764770 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:58.764753 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:29:59.780258 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:59.780212 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" event={"ID":"b4076516-c2ce-4d4c-b6de-78c94c179e41","Type":"ContainerStarted","Data":"ba6791c5e71a9dcccab790b7cd24013eb8fa7294772f83163dbc51115f63adc2"} Sep 08 16:29:59.922336 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:59.922287 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:29:59.922611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:59.922586 2555 patch_prober.go:28] interesting pod/e2e-autores-293166-openshift-default-599d8647fb-gp64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 08 16:29:59.922669 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:29:59.922637 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 08 16:30:00.922472 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:00.922437 2555 patch_prober.go:28] interesting pod/e2e-autores-293166-openshift-default-599d8647fb-gp64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 08 16:30:00.922902 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:00.922499 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 08 16:30:01.922452 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:01.922410 2555 patch_prober.go:28] interesting pod/e2e-autores-293166-openshift-default-599d8647fb-gp64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 08 16:30:01.922638 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:01.922476 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 08 16:30:02.922215 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:02.922181 2555 patch_prober.go:28] interesting pod/e2e-autores-293166-openshift-default-599d8647fb-gp64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 08 16:30:02.922496 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:02.922464 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 08 16:30:03.923339 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:03.923300 2555 patch_prober.go:28] interesting pod/e2e-autores-293166-openshift-default-599d8647fb-gp64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 08 16:30:03.923760 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:03.923358 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 08 16:30:04.922773 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:04.922720 2555 patch_prober.go:28] interesting pod/e2e-autores-293166-openshift-default-599d8647fb-gp64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 08 16:30:04.923007 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:04.922794 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 08 16:30:05.927001 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:05.926967 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:30:05.927520 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:05.927239 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:30:05.928109 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:05.928091 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:30:19.478471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.478412 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn"] Sep 08 16:30:19.478919 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.478654 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" podUID="94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" containerName="maas-api" containerID="cri-o://541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb" gracePeriod=30 Sep 08 16:30:19.742623 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.742524 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:30:19.841758 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.841713 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hhd4\" (UniqueName: \"kubernetes.io/projected/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-kube-api-access-9hhd4\") pod \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " Sep 08 16:30:19.841938 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.841867 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-metrics-certs\") pod \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " Sep 08 16:30:19.841938 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.841896 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-maas-api-tls\") pod \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " Sep 08 16:30:19.841938 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.841925 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-openshift-service-ca\") pod \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\" (UID: \"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd\") " Sep 08 16:30:19.842413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.842364 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" (UID: "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:30:19.844548 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.844521 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-kube-api-access-9hhd4" (OuterVolumeSpecName: "kube-api-access-9hhd4") pod "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" (UID: "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd"). InnerVolumeSpecName "kube-api-access-9hhd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:30:19.844653 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.844557 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" (UID: "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:30:19.844802 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.844773 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" (UID: "94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:30:19.901965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.901922 2555 generic.go:356] "Generic (PLEG): container finished" podID="94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" containerID="541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb" exitCode=0 Sep 08 16:30:19.902163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.901989 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" event={"ID":"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd","Type":"ContainerDied","Data":"541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb"} Sep 08 16:30:19.902163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.902017 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" Sep 08 16:30:19.902163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.902054 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn" event={"ID":"94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd","Type":"ContainerDied","Data":"1e83da9045d114bd5f8e59041b8a52dffc315ad4e33034b3395ed06adc5298b1"} Sep 08 16:30:19.902163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.902089 2555 scope.go:117] "RemoveContainer" containerID="541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb" Sep 08 16:30:19.916347 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.916291 2555 scope.go:117] "RemoveContainer" containerID="541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb" Sep 08 16:30:19.917068 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:30:19.917025 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb\": container with ID starting with 541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb not found: ID does not exist" containerID="541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb" Sep 08 16:30:19.917238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.917206 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb"} err="failed to get container status \"541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb\": rpc error: code = NotFound desc = could not find container \"541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb\": container with ID starting with 541cf6d7e834c247668ee7a3702cc8d26a1ae476f5ddd051d797156dcf27debb not found: ID does not exist" Sep 08 16:30:19.943307 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.943274 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:19.943307 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.943302 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:19.943515 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.943314 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:19.943515 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.943334 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhd4\" (UniqueName: \"kubernetes.io/projected/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd-kube-api-access-9hhd4\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:19.947059 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.947022 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn"] Sep 08 16:30:19.957193 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:19.957157 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec338109-59b6b55d7c-klxbn"] Sep 08 16:30:20.909239 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.909198 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb"] Sep 08 16:30:20.909833 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.909812 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" containerName="maas-api" Sep 08 16:30:20.909925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.909836 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" containerName="maas-api" Sep 08 16:30:20.909978 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.909933 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" containerName="maas-api" Sep 08 16:30:20.914629 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.914605 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.918822 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.918795 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8f31ec88-gw-openshift-default-dockercfg-ndm5f\"" Sep 08 16:30:20.938105 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.938070 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd" path="/var/lib/kubelet/pods/94d2d3f0-36be-47a9-9d9a-abd2bf92f3cd/volumes" Sep 08 16:30:20.938385 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.938369 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb"] Sep 08 16:30:20.954314 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954275 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-data\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954330 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-istio-token\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954400 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879f28fe-b9a4-4320-98d6-804f486887f7-istiod-ca-cert\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954542 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954508 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-socket\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954578 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954549 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-credential-socket\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954610 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954583 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-envoy\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954652 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-certs\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954689 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954649 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879f28fe-b9a4-4320-98d6-804f486887f7-istio-podinfo\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:20.954689 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:20.954665 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhk9\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-kube-api-access-mwhk9\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.055829 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.055784 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-credential-socket\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.055849 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-envoy\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.055878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-certs\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.055923 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879f28fe-b9a4-4320-98d6-804f486887f7-istio-podinfo\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.055949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhk9\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-kube-api-access-mwhk9\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.055991 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-data\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056030 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-istio-token\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056066 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879f28fe-b9a4-4320-98d6-804f486887f7-istiod-ca-cert\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056158 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-socket\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056302 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-credential-socket\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056328 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-certs\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056407 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-data\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056689 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056666 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-socket\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.056934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.056912 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879f28fe-b9a4-4320-98d6-804f486887f7-istiod-ca-cert\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.058447 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.058426 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-envoy\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.058554 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.058510 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879f28fe-b9a4-4320-98d6-804f486887f7-istio-podinfo\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.074019 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.073991 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhk9\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-kube-api-access-mwhk9\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.074648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.074626 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-istio-token\") pod \"e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.225238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.225143 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:21.508374 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.508348 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb"] Sep 08 16:30:21.509909 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:30:21.509870 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879f28fe_b9a4_4320_98d6_804f486887f7.slice/crio-74918defbf77f735d5affeef1ebcee0e756bc79e761f2fe7aea26fcb6da157a7 WatchSource:0}: Error finding container 74918defbf77f735d5affeef1ebcee0e756bc79e761f2fe7aea26fcb6da157a7: Status 404 returned error can't find the container with id 74918defbf77f735d5affeef1ebcee0e756bc79e761f2fe7aea26fcb6da157a7 Sep 08 16:30:21.512219 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.512187 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:30:21.512332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.512304 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:30:21.512375 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.512342 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:30:21.919649 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.919608 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" event={"ID":"879f28fe-b9a4-4320-98d6-804f486887f7","Type":"ContainerStarted","Data":"4e6fda3da446c9d2ce1c61807e21cb7d585c5822976ff84ba7b9d8ff9d5795b6"} Sep 08 16:30:21.920093 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.919672 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" event={"ID":"879f28fe-b9a4-4320-98d6-804f486887f7","Type":"ContainerStarted","Data":"74918defbf77f735d5affeef1ebcee0e756bc79e761f2fe7aea26fcb6da157a7"} Sep 08 16:30:21.943294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:21.943220 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podStartSLOduration=1.943202173 podStartE2EDuration="1.943202173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:21.941984369 +0000 UTC m=+1055.536983152" watchObservedRunningTime="2026-09-08 16:30:21.943202173 +0000 UTC m=+1055.538200998" Sep 08 16:30:22.226106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:22.226008 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:22.231500 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:22.231471 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:22.925759 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:22.925727 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:22.926799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:22.926769 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:23.707769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.707727 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v"] Sep 08 16:30:23.711562 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.711543 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.713815 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.713788 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8f31ec88-serving-cert\"" Sep 08 16:30:23.721126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.721098 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v"] Sep 08 16:30:23.786027 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.785987 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ff185-e9e2-432b-831e-b470c7fac156-openshift-service-ca\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.786189 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.786065 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mrr5\" (UniqueName: \"kubernetes.io/projected/6d7ff185-e9e2-432b-831e-b470c7fac156-kube-api-access-8mrr5\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.786189 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.786135 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.786189 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.786168 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-metrics-certs\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.887606 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.887568 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ff185-e9e2-432b-831e-b470c7fac156-openshift-service-ca\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.887606 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.887606 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mrr5\" (UniqueName: \"kubernetes.io/projected/6d7ff185-e9e2-432b-831e-b470c7fac156-kube-api-access-8mrr5\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.887885 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.887689 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.887885 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.887728 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-metrics-certs\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.887885 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:30:23.887851 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-serving-cert: secret "maas-api-e2e-migrate-8f31ec88-serving-cert" not found Sep 08 16:30:23.888009 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:30:23.887928 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls podName:6d7ff185-e9e2-432b-831e-b470c7fac156 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:24.38790602 +0000 UTC m=+1057.982904780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls") pod "maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" (UID: "6d7ff185-e9e2-432b-831e-b470c7fac156") : secret "maas-api-e2e-migrate-8f31ec88-serving-cert" not found Sep 08 16:30:23.888376 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.888355 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ff185-e9e2-432b-831e-b470c7fac156-openshift-service-ca\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.890203 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.890184 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-metrics-certs\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:23.901817 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:23.901780 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mrr5\" (UniqueName: \"kubernetes.io/projected/6d7ff185-e9e2-432b-831e-b470c7fac156-kube-api-access-8mrr5\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:24.392424 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:24.392383 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:24.394985 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:24.394952 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls\") pod \"maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:24.624194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:24.624161 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:24.903978 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:24.903948 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v"] Sep 08 16:30:24.905493 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:30:24.905466 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7ff185_e9e2_432b_831e_b470c7fac156.slice/crio-bc193474a76f3e7d4cda85e5f49cb2edf695d96b4b2a96ea4ef6f30b8a6ca17b WatchSource:0}: Error finding container bc193474a76f3e7d4cda85e5f49cb2edf695d96b4b2a96ea4ef6f30b8a6ca17b: Status 404 returned error can't find the container with id bc193474a76f3e7d4cda85e5f49cb2edf695d96b4b2a96ea4ef6f30b8a6ca17b Sep 08 16:30:24.939433 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:24.939346 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" event={"ID":"6d7ff185-e9e2-432b-831e-b470c7fac156","Type":"ContainerStarted","Data":"bc193474a76f3e7d4cda85e5f49cb2edf695d96b4b2a96ea4ef6f30b8a6ca17b"} Sep 08 16:30:25.946942 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:25.946905 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" event={"ID":"6d7ff185-e9e2-432b-831e-b470c7fac156","Type":"ContainerStarted","Data":"3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160"} Sep 08 16:30:25.947365 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:25.946977 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:25.966921 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:25.966863 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" podStartSLOduration=2.510544155 podStartE2EDuration="2.966844743s" totalImagesPullingTime="456.300588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:25.964189144 +0000 UTC m=+1059.559187923" watchObservedRunningTime="2026-09-08 16:30:25.966844743 +0000 UTC m=+1059.561843501" Sep 08 16:30:31.957843 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:31.957805 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:43.049709 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:43.049613 2555 generic.go:356] "Generic (PLEG): container finished" podID="879f28fe-b9a4-4320-98d6-804f486887f7" containerID="4e6fda3da446c9d2ce1c61807e21cb7d585c5822976ff84ba7b9d8ff9d5795b6" exitCode=0 Sep 08 16:30:43.050136 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:43.049698 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" event={"ID":"879f28fe-b9a4-4320-98d6-804f486887f7","Type":"ContainerDied","Data":"4e6fda3da446c9d2ce1c61807e21cb7d585c5822976ff84ba7b9d8ff9d5795b6"} Sep 08 16:30:43.050202 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:43.050172 2555 scope.go:117] "RemoveContainer" containerID="4e6fda3da446c9d2ce1c61807e21cb7d585c5822976ff84ba7b9d8ff9d5795b6" Sep 08 16:30:43.051377 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:43.051343 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:30:43.051514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:43.051434 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:30:43.051514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:43.051486 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:30:44.068406 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:44.068364 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" event={"ID":"879f28fe-b9a4-4320-98d6-804f486887f7","Type":"ContainerStarted","Data":"d89c8e106e840c65a1461be3485fbacdb4e935223117674237750f049475d6b5"} Sep 08 16:30:44.225659 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:44.225613 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:44.225891 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:44.225865 2555 patch_prober.go:28] interesting pod/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 08 16:30:44.225948 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:44.225929 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 08 16:30:45.225876 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:45.225838 2555 patch_prober.go:28] interesting pod/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 08 16:30:45.226299 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:45.225910 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 08 16:30:46.226304 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:46.226231 2555 patch_prober.go:28] interesting pod/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 08 16:30:46.226723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:46.226335 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 08 16:30:47.226534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:47.226440 2555 patch_prober.go:28] interesting pod/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 08 16:30:47.226534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:47.226506 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 08 16:30:48.226426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:48.226384 2555 patch_prober.go:28] interesting pod/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 08 16:30:48.226611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:48.226447 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 08 16:30:49.225985 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:49.225950 2555 patch_prober.go:28] interesting pod/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 08 16:30:49.226177 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:49.226007 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 08 16:30:50.230515 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:50.230481 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:50.230941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:50.230743 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:50.231404 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:50.231385 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:30:54.607746 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.607711 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v"] Sep 08 16:30:54.608226 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.607967 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" podUID="6d7ff185-e9e2-432b-831e-b470c7fac156" containerName="maas-api" containerID="cri-o://3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160" gracePeriod=30 Sep 08 16:30:54.853307 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.853276 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:54.993707 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.993612 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-metrics-certs\") pod \"6d7ff185-e9e2-432b-831e-b470c7fac156\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " Sep 08 16:30:54.993884 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.993734 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ff185-e9e2-432b-831e-b470c7fac156-openshift-service-ca\") pod \"6d7ff185-e9e2-432b-831e-b470c7fac156\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " Sep 08 16:30:54.993884 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.993803 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mrr5\" (UniqueName: \"kubernetes.io/projected/6d7ff185-e9e2-432b-831e-b470c7fac156-kube-api-access-8mrr5\") pod \"6d7ff185-e9e2-432b-831e-b470c7fac156\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " Sep 08 16:30:54.993884 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.993850 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls\") pod \"6d7ff185-e9e2-432b-831e-b470c7fac156\" (UID: \"6d7ff185-e9e2-432b-831e-b470c7fac156\") " Sep 08 16:30:54.994393 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.994361 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7ff185-e9e2-432b-831e-b470c7fac156-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d7ff185-e9e2-432b-831e-b470c7fac156" (UID: "6d7ff185-e9e2-432b-831e-b470c7fac156"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:30:54.996446 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.996400 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d7ff185-e9e2-432b-831e-b470c7fac156" (UID: "6d7ff185-e9e2-432b-831e-b470c7fac156"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:30:54.996670 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.996617 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7ff185-e9e2-432b-831e-b470c7fac156-kube-api-access-8mrr5" (OuterVolumeSpecName: "kube-api-access-8mrr5") pod "6d7ff185-e9e2-432b-831e-b470c7fac156" (UID: "6d7ff185-e9e2-432b-831e-b470c7fac156"). InnerVolumeSpecName "kube-api-access-8mrr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:30:54.997333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:54.997231 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d7ff185-e9e2-432b-831e-b470c7fac156" (UID: "6d7ff185-e9e2-432b-831e-b470c7fac156"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:30:55.021718 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.021666 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb"] Sep 08 16:30:55.022228 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.022012 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" containerID="cri-o://d89c8e106e840c65a1461be3485fbacdb4e935223117674237750f049475d6b5" gracePeriod=30 Sep 08 16:30:55.095577 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.095540 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:55.095577 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.095566 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ff185-e9e2-432b-831e-b470c7fac156-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:55.095577 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.095576 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ff185-e9e2-432b-831e-b470c7fac156-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:55.095577 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.095585 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mrr5\" (UniqueName: \"kubernetes.io/projected/6d7ff185-e9e2-432b-831e-b470c7fac156-kube-api-access-8mrr5\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:30:55.140739 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.140700 2555 generic.go:356] "Generic (PLEG): container finished" podID="6d7ff185-e9e2-432b-831e-b470c7fac156" containerID="3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160" exitCode=0 Sep 08 16:30:55.140739 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.140747 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" Sep 08 16:30:55.141004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.140777 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" event={"ID":"6d7ff185-e9e2-432b-831e-b470c7fac156","Type":"ContainerDied","Data":"3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160"} Sep 08 16:30:55.141004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.140834 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v" event={"ID":"6d7ff185-e9e2-432b-831e-b470c7fac156","Type":"ContainerDied","Data":"bc193474a76f3e7d4cda85e5f49cb2edf695d96b4b2a96ea4ef6f30b8a6ca17b"} Sep 08 16:30:55.141004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.140857 2555 scope.go:117] "RemoveContainer" containerID="3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160" Sep 08 16:30:55.152809 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.152783 2555 scope.go:117] "RemoveContainer" containerID="3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160" Sep 08 16:30:55.153315 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:30:55.153292 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160\": container with ID starting with 3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160 not found: ID does not exist" containerID="3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160" Sep 08 16:30:55.153398 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.153327 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160"} err="failed to get container status \"3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160\": rpc error: code = NotFound desc = could not find container \"3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160\": container with ID starting with 3facc7026d622c54082399a8b43127ff41187862a54a4bf64ca39fc8cf443160 not found: ID does not exist" Sep 08 16:30:55.175396 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.175357 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v"] Sep 08 16:30:55.184471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:55.184435 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f31ec88-848d98df66-2zd5v"] Sep 08 16:30:56.938656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:56.938618 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d7ff185-e9e2-432b-831e-b470c7fac156" path="/var/lib/kubelet/pods/6d7ff185-e9e2-432b-831e-b470c7fac156/volumes" Sep 08 16:30:59.116511 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.116470 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68"] Sep 08 16:30:59.117187 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.117163 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7ff185-e9e2-432b-831e-b470c7fac156" containerName="maas-api" Sep 08 16:30:59.117286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.117192 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7ff185-e9e2-432b-831e-b470c7fac156" containerName="maas-api" Sep 08 16:30:59.117349 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.117337 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7ff185-e9e2-432b-831e-b470c7fac156" containerName="maas-api" Sep 08 16:30:59.121010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.120984 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.123976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.123945 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-aeb37378-serving-cert\"" Sep 08 16:30:59.131539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.131505 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0dc936-7ffb-4bcd-96a0-0c206394a907-openshift-service-ca\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.131677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.131576 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.131677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.131622 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkq2\" (UniqueName: \"kubernetes.io/projected/5a0dc936-7ffb-4bcd-96a0-0c206394a907-kube-api-access-4mkq2\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.131785 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.131680 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-metrics-certs\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.232363 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.232320 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0dc936-7ffb-4bcd-96a0-0c206394a907-openshift-service-ca\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.232567 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.232376 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.232567 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.232396 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkq2\" (UniqueName: \"kubernetes.io/projected/5a0dc936-7ffb-4bcd-96a0-0c206394a907-kube-api-access-4mkq2\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.232567 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.232424 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-metrics-certs\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.232567 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:30:59.232537 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-serving-cert: secret "maas-api-e2e-ait-aeb37378-serving-cert" not found Sep 08 16:30:59.232794 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:30:59.232608 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls podName:5a0dc936-7ffb-4bcd-96a0-0c206394a907 nodeName:}" failed. No retries permitted until 2026-09-08 16:30:59.732589616 +0000 UTC m=+1093.327588359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls") pod "maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" (UID: "5a0dc936-7ffb-4bcd-96a0-0c206394a907") : secret "maas-api-e2e-ait-aeb37378-serving-cert" not found Sep 08 16:30:59.233014 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.232993 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0dc936-7ffb-4bcd-96a0-0c206394a907-openshift-service-ca\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.234935 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.234911 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-metrics-certs\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.252934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.252895 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkq2\" (UniqueName: \"kubernetes.io/projected/5a0dc936-7ffb-4bcd-96a0-0c206394a907-kube-api-access-4mkq2\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.257213 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.257178 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68"] Sep 08 16:30:59.737689 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.737646 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:30:59.740189 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:30:59.740160 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls\") pod \"maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:31:00.032590 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.032502 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:31:00.173388 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.173353 2555 generic.go:356] "Generic (PLEG): container finished" podID="879f28fe-b9a4-4320-98d6-804f486887f7" containerID="d89c8e106e840c65a1461be3485fbacdb4e935223117674237750f049475d6b5" exitCode=0 Sep 08 16:31:00.173861 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.173400 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" event={"ID":"879f28fe-b9a4-4320-98d6-804f486887f7","Type":"ContainerDied","Data":"d89c8e106e840c65a1461be3485fbacdb4e935223117674237750f049475d6b5"} Sep 08 16:31:00.173861 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.173459 2555 scope.go:117] "RemoveContainer" containerID="4e6fda3da446c9d2ce1c61807e21cb7d585c5822976ff84ba7b9d8ff9d5795b6" Sep 08 16:31:00.304774 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.304747 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:31:00.321758 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.321726 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68"] Sep 08 16:31:00.322154 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:31:00.322118 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0dc936_7ffb_4bcd_96a0_0c206394a907.slice/crio-382527cec21f3abae133d5a94ed8b03a86c07e24786faf2c619ee00ed54e74c6 WatchSource:0}: Error finding container 382527cec21f3abae133d5a94ed8b03a86c07e24786faf2c619ee00ed54e74c6: Status 404 returned error can't find the container with id 382527cec21f3abae133d5a94ed8b03a86c07e24786faf2c619ee00ed54e74c6 Sep 08 16:31:00.343232 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343154 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-certs\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343232 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343225 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879f28fe-b9a4-4320-98d6-804f486887f7-istio-podinfo\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343293 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-socket\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343334 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-data\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343390 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-envoy\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343437 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879f28fe-b9a4-4320-98d6-804f486887f7-istiod-ca-cert\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343484 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-istio-token\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343512 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-credential-socket\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.343856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343572 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwhk9\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-kube-api-access-mwhk9\") pod \"879f28fe-b9a4-4320-98d6-804f486887f7\" (UID: \"879f28fe-b9a4-4320-98d6-804f486887f7\") " Sep 08 16:31:00.344529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.343940 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879f28fe-b9a4-4320-98d6-804f486887f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:31:00.344529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.344213 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:31:00.344529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.344327 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:31:00.344529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.344495 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:31:00.344529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.344512 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:31:00.346780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.346737 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/879f28fe-b9a4-4320-98d6-804f486887f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:31:00.346908 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.346854 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-kube-api-access-mwhk9" (OuterVolumeSpecName: "kube-api-access-mwhk9") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "kube-api-access-mwhk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:31:00.347010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.346973 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:31:00.347129 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.347059 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "879f28fe-b9a4-4320-98d6-804f486887f7" (UID: "879f28fe-b9a4-4320-98d6-804f486887f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:31:00.444693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444650 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879f28fe-b9a4-4320-98d6-804f486887f7-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444686 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444694 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444704 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhk9\" (UniqueName: \"kubernetes.io/projected/879f28fe-b9a4-4320-98d6-804f486887f7-kube-api-access-mwhk9\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444973 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444713 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444973 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444721 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879f28fe-b9a4-4320-98d6-804f486887f7-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444973 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444729 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444973 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444737 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:00.444973 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:00.444745 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879f28fe-b9a4-4320-98d6-804f486887f7-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:01.181283 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.181236 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" Sep 08 16:31:01.181283 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.181261 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb" event={"ID":"879f28fe-b9a4-4320-98d6-804f486887f7","Type":"ContainerDied","Data":"74918defbf77f735d5affeef1ebcee0e756bc79e761f2fe7aea26fcb6da157a7"} Sep 08 16:31:01.181833 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.181328 2555 scope.go:117] "RemoveContainer" containerID="d89c8e106e840c65a1461be3485fbacdb4e935223117674237750f049475d6b5" Sep 08 16:31:01.183119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.183088 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" event={"ID":"5a0dc936-7ffb-4bcd-96a0-0c206394a907","Type":"ContainerStarted","Data":"5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876"} Sep 08 16:31:01.183280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.183149 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" event={"ID":"5a0dc936-7ffb-4bcd-96a0-0c206394a907","Type":"ContainerStarted","Data":"382527cec21f3abae133d5a94ed8b03a86c07e24786faf2c619ee00ed54e74c6"} Sep 08 16:31:01.183369 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.183346 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:31:01.203352 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.203309 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb"] Sep 08 16:31:01.206130 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.206091 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8f31ec88-gw-openshift-default-7466dd4c7b-5b2mb"] Sep 08 16:31:01.227972 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:01.227910 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" podStartSLOduration=1.8394432269999998 podStartE2EDuration="2.227891034s" totalImagesPullingTime="388.447807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:01.225329686 +0000 UTC m=+1094.820328468" watchObservedRunningTime="2026-09-08 16:31:01.227891034 +0000 UTC m=+1094.822889892" Sep 08 16:31:02.939561 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:02.939522 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" path="/var/lib/kubelet/pods/879f28fe-b9a4-4320-98d6-804f486887f7/volumes" Sep 08 16:31:07.196081 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:07.196048 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:31:28.777916 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.777878 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf"] Sep 08 16:31:28.778535 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.778516 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" Sep 08 16:31:28.778601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.778537 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" Sep 08 16:31:28.778601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.778561 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" Sep 08 16:31:28.778601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.778571 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" Sep 08 16:31:28.778697 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.778688 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" Sep 08 16:31:28.782639 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.782616 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:28.786568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.786223 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-293166\"/\"autores-model-293166-kserve-self-signed-certs\"" Sep 08 16:31:28.786568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.786432 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-293166\"/\"default-dockercfg-wmh7h\"" Sep 08 16:31:28.786568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.786223 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-293166\"/\"openshift-service-ca.crt\"" Sep 08 16:31:28.786848 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.786223 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-293166\"/\"kube-root-ca.crt\"" Sep 08 16:31:28.790996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.790963 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf"] Sep 08 16:31:28.932369 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.932326 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-model-cache\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:28.932569 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.932399 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-home\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:28.932569 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.932441 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2wn\" (UniqueName: \"kubernetes.io/projected/d2d64360-4feb-4a31-9e65-b2493514a2de-kube-api-access-qd2wn\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:28.932569 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.932475 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-tmp-dir\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:28.932711 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.932568 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d64360-4feb-4a31-9e65-b2493514a2de-tls-certs\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:28.932711 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:28.932617 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-dshm\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.033754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.033656 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-home\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.033754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.033696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2wn\" (UniqueName: \"kubernetes.io/projected/d2d64360-4feb-4a31-9e65-b2493514a2de-kube-api-access-qd2wn\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.033754 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.033723 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-tmp-dir\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.034068 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.033783 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d64360-4feb-4a31-9e65-b2493514a2de-tls-certs\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.034068 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.033819 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-dshm\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.034068 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.033868 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-model-cache\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.034217 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.034154 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-home\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.034217 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.034180 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-tmp-dir\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.034323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.034223 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-model-cache\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.036129 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.036098 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-dshm\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.036415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.036398 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d64360-4feb-4a31-9e65-b2493514a2de-tls-certs\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.042385 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.042358 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2wn\" (UniqueName: \"kubernetes.io/projected/d2d64360-4feb-4a31-9e65-b2493514a2de-kube-api-access-qd2wn\") pod \"autores-model-293166-kserve-c945478d7-jfvkf\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.096447 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.096399 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:29.372782 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.372751 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf"] Sep 08 16:31:29.374635 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:31:29.374605 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2d64360_4feb_4a31_9e65_b2493514a2de.slice/crio-7c3aa35a4605ba6935e40801175734ccf10df96dc9c57f2b9788ccc9ae3d491a WatchSource:0}: Error finding container 7c3aa35a4605ba6935e40801175734ccf10df96dc9c57f2b9788ccc9ae3d491a: Status 404 returned error can't find the container with id 7c3aa35a4605ba6935e40801175734ccf10df96dc9c57f2b9788ccc9ae3d491a Sep 08 16:31:29.759202 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.759102 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68"] Sep 08 16:31:29.759387 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:29.759351 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" podUID="5a0dc936-7ffb-4bcd-96a0-0c206394a907" containerName="maas-api" containerID="cri-o://5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876" gracePeriod=30 Sep 08 16:31:30.013539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.013458 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:31:30.144040 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.143995 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0dc936-7ffb-4bcd-96a0-0c206394a907-openshift-service-ca\") pod \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " Sep 08 16:31:30.144292 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.144055 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-metrics-certs\") pod \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " Sep 08 16:31:30.144292 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.144165 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls\") pod \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " Sep 08 16:31:30.144292 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.144204 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mkq2\" (UniqueName: \"kubernetes.io/projected/5a0dc936-7ffb-4bcd-96a0-0c206394a907-kube-api-access-4mkq2\") pod \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\" (UID: \"5a0dc936-7ffb-4bcd-96a0-0c206394a907\") " Sep 08 16:31:30.144476 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.144439 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0dc936-7ffb-4bcd-96a0-0c206394a907-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a0dc936-7ffb-4bcd-96a0-0c206394a907" (UID: "5a0dc936-7ffb-4bcd-96a0-0c206394a907"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:31:30.144596 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.144566 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0dc936-7ffb-4bcd-96a0-0c206394a907-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:30.146804 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.146757 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a0dc936-7ffb-4bcd-96a0-0c206394a907" (UID: "5a0dc936-7ffb-4bcd-96a0-0c206394a907"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:31:30.147096 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.147064 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0dc936-7ffb-4bcd-96a0-0c206394a907-kube-api-access-4mkq2" (OuterVolumeSpecName: "kube-api-access-4mkq2") pod "5a0dc936-7ffb-4bcd-96a0-0c206394a907" (UID: "5a0dc936-7ffb-4bcd-96a0-0c206394a907"). InnerVolumeSpecName "kube-api-access-4mkq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:31:30.147540 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.147505 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a0dc936-7ffb-4bcd-96a0-0c206394a907" (UID: "5a0dc936-7ffb-4bcd-96a0-0c206394a907"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:31:30.246143 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.246103 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:30.246143 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.246141 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mkq2\" (UniqueName: \"kubernetes.io/projected/5a0dc936-7ffb-4bcd-96a0-0c206394a907-kube-api-access-4mkq2\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:30.246143 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.246153 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0dc936-7ffb-4bcd-96a0-0c206394a907-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:31:30.354231 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.354186 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" event={"ID":"d2d64360-4feb-4a31-9e65-b2493514a2de","Type":"ContainerStarted","Data":"57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a"} Sep 08 16:31:30.354461 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.354271 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" event={"ID":"d2d64360-4feb-4a31-9e65-b2493514a2de","Type":"ContainerStarted","Data":"7c3aa35a4605ba6935e40801175734ccf10df96dc9c57f2b9788ccc9ae3d491a"} Sep 08 16:31:30.355646 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.355619 2555 generic.go:356] "Generic (PLEG): container finished" podID="5a0dc936-7ffb-4bcd-96a0-0c206394a907" containerID="5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876" exitCode=0 Sep 08 16:31:30.355751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.355672 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" Sep 08 16:31:30.355751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.355690 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" event={"ID":"5a0dc936-7ffb-4bcd-96a0-0c206394a907","Type":"ContainerDied","Data":"5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876"} Sep 08 16:31:30.355822 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.355749 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68" event={"ID":"5a0dc936-7ffb-4bcd-96a0-0c206394a907","Type":"ContainerDied","Data":"382527cec21f3abae133d5a94ed8b03a86c07e24786faf2c619ee00ed54e74c6"} Sep 08 16:31:30.355822 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.355778 2555 scope.go:117] "RemoveContainer" containerID="5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876" Sep 08 16:31:30.366315 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.366286 2555 scope.go:117] "RemoveContainer" containerID="5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876" Sep 08 16:31:30.366651 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:31:30.366632 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876\": container with ID starting with 5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876 not found: ID does not exist" containerID="5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876" Sep 08 16:31:30.366707 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.366661 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876"} err="failed to get container status \"5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876\": rpc error: code = NotFound desc = could not find container \"5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876\": container with ID starting with 5ca75fb619c88c015ba1b0d86332e24a30c298afa909e6e29935d3269726c876 not found: ID does not exist" Sep 08 16:31:30.387801 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.387744 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" podStartSLOduration=2.387725642 podStartE2EDuration="2.387725642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:30.385451242 +0000 UTC m=+1123.980450001" watchObservedRunningTime="2026-09-08 16:31:30.387725642 +0000 UTC m=+1123.982724439" Sep 08 16:31:30.414294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.414240 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68"] Sep 08 16:31:30.424420 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.424387 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aeb37378-66c9dfbf99-twv68"] Sep 08 16:31:30.938001 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:30.937962 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a0dc936-7ffb-4bcd-96a0-0c206394a907" path="/var/lib/kubelet/pods/5a0dc936-7ffb-4bcd-96a0-0c206394a907/volumes" Sep 08 16:31:32.145359 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.145321 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62"] Sep 08 16:31:32.145767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.145754 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0dc936-7ffb-4bcd-96a0-0c206394a907" containerName="maas-api" Sep 08 16:31:32.145767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.145768 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0dc936-7ffb-4bcd-96a0-0c206394a907" containerName="maas-api" Sep 08 16:31:32.145848 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.145841 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0dc936-7ffb-4bcd-96a0-0c206394a907" containerName="maas-api" Sep 08 16:31:32.145883 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.145850 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="879f28fe-b9a4-4320-98d6-804f486887f7" containerName="istio-proxy" Sep 08 16:31:32.150709 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.150683 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.153271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.153229 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1b78b587-openshift-default-dockercfg-gjxxc\"" Sep 08 16:31:32.164558 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.164526 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62"] Sep 08 16:31:32.264559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264519 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istiod-ca-cert\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.264779 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264664 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-token\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.264779 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264737 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-certs\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.264779 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264762 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwxvf\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-kube-api-access-rwxvf\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.264931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264802 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-data\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.264931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264842 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-envoy\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.264931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264869 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-socket\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.265054 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264949 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-credential-socket\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.265054 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.264999 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-podinfo\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366272 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-token\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366465 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366296 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-certs\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366465 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366316 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwxvf\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-kube-api-access-rwxvf\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366585 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366495 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-data\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366585 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366558 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-envoy\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366688 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366602 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-socket\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366799 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366778 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-credential-socket\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366861 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-podinfo\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366998 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istiod-ca-cert\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.366998 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.366975 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-socket\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.367085 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.367021 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-certs\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.367085 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.367058 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-credential-socket\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.367291 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.367272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-data\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.367686 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.367666 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istiod-ca-cert\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.368916 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.368885 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-envoy\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.369326 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.369307 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-podinfo\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.375107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.375084 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-token\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.375107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.375091 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwxvf\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-kube-api-access-rwxvf\") pod \"e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.462003 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.461906 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:32.742976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.742941 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62"] Sep 08 16:31:32.743933 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:31:32.743905 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ef8e10_5815_4175_ab7c_da4f50fe1116.slice/crio-ceed8a62c9b2b946a45b7bf45aa94045ee0b876fa086047e43ac6ab62d62a11a WatchSource:0}: Error finding container ceed8a62c9b2b946a45b7bf45aa94045ee0b876fa086047e43ac6ab62d62a11a: Status 404 returned error can't find the container with id ceed8a62c9b2b946a45b7bf45aa94045ee0b876fa086047e43ac6ab62d62a11a Sep 08 16:31:32.746567 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.746522 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:31:32.746662 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.746633 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:31:32.746703 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:32.746680 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:31:33.376715 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:33.376678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" event={"ID":"a0ef8e10-5815-4175-ab7c-da4f50fe1116","Type":"ContainerStarted","Data":"f6373a68adda7826bc82dc19154a2428f2b312869d8547ab65f85b2bb04f468f"} Sep 08 16:31:33.377171 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:33.376745 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" event={"ID":"a0ef8e10-5815-4175-ab7c-da4f50fe1116","Type":"ContainerStarted","Data":"ceed8a62c9b2b946a45b7bf45aa94045ee0b876fa086047e43ac6ab62d62a11a"} Sep 08 16:31:33.398102 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:33.398037 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" podStartSLOduration=1.398016839 podStartE2EDuration="1.398016839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:33.396848982 +0000 UTC m=+1126.991847752" watchObservedRunningTime="2026-09-08 16:31:33.398016839 +0000 UTC m=+1126.993015598" Sep 08 16:31:33.462346 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:33.462292 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:33.467721 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:33.467689 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:34.383099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:34.383062 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:34.384112 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:34.384087 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:39.096597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:39.096537 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:39.096597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:39.096608 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:39.110363 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:39.110332 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:39.428217 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:39.428133 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:31:56.533783 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:56.533697 2555 generic.go:356] "Generic (PLEG): container finished" podID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerID="f6373a68adda7826bc82dc19154a2428f2b312869d8547ab65f85b2bb04f468f" exitCode=0 Sep 08 16:31:56.533783 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:56.533769 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" event={"ID":"a0ef8e10-5815-4175-ab7c-da4f50fe1116","Type":"ContainerDied","Data":"f6373a68adda7826bc82dc19154a2428f2b312869d8547ab65f85b2bb04f468f"} Sep 08 16:31:56.534233 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:56.534180 2555 scope.go:117] "RemoveContainer" containerID="f6373a68adda7826bc82dc19154a2428f2b312869d8547ab65f85b2bb04f468f" Sep 08 16:31:56.535420 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:56.535389 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:31:56.535496 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:56.535479 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:31:56.535540 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:56.535519 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:31:57.544140 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:57.544088 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" event={"ID":"a0ef8e10-5815-4175-ab7c-da4f50fe1116","Type":"ContainerStarted","Data":"af1eac65d346024f9d1bbe6f7617804d476089e5894e535cf47a08818a0c2e19"} Sep 08 16:31:58.462626 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:58.462580 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:31:58.462862 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:58.462743 2555 patch_prober.go:28] interesting pod/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 08 16:31:58.462862 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:58.462801 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 08 16:31:59.463176 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:59.463135 2555 patch_prober.go:28] interesting pod/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 08 16:31:59.463850 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:31:59.463813 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 08 16:32:00.462762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:00.462710 2555 patch_prober.go:28] interesting pod/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 08 16:32:00.463105 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:00.462806 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 08 16:32:01.462956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:01.462914 2555 patch_prober.go:28] interesting pod/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 08 16:32:01.463625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:01.463591 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 08 16:32:02.462549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:02.462507 2555 patch_prober.go:28] interesting pod/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 08 16:32:02.462887 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:02.462854 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 08 16:32:03.466617 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:03.466577 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:32:03.467059 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:03.466881 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:32:03.467750 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:03.467729 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:32:07.480030 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:07.479995 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62"] Sep 08 16:32:07.480463 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:07.480201 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" containerID="cri-o://af1eac65d346024f9d1bbe6f7617804d476089e5894e535cf47a08818a0c2e19" gracePeriod=30 Sep 08 16:32:12.643087 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.643053 2555 generic.go:356] "Generic (PLEG): container finished" podID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerID="af1eac65d346024f9d1bbe6f7617804d476089e5894e535cf47a08818a0c2e19" exitCode=0 Sep 08 16:32:12.643536 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.643131 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" event={"ID":"a0ef8e10-5815-4175-ab7c-da4f50fe1116","Type":"ContainerDied","Data":"af1eac65d346024f9d1bbe6f7617804d476089e5894e535cf47a08818a0c2e19"} Sep 08 16:32:12.643536 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.643200 2555 scope.go:117] "RemoveContainer" containerID="f6373a68adda7826bc82dc19154a2428f2b312869d8547ab65f85b2bb04f468f" Sep 08 16:32:12.742345 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.742270 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:32:12.799858 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.799816 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istiod-ca-cert\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800067 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.799893 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-socket\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800067 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.799930 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-data\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800067 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.799968 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwxvf\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-kube-api-access-rwxvf\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800067 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800001 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-podinfo\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800067 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800027 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-credential-socket\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800067 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800060 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-envoy\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800118 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-certs\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800200 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-token\") pod \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\" (UID: \"a0ef8e10-5815-4175-ab7c-da4f50fe1116\") " Sep 08 16:32:12.800413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800230 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:32:12.800413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800284 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-data" (OuterVolumeSpecName: "istio-data") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:32:12.800620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800493 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:32:12.800620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800516 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:32:12.800620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800559 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.800620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800573 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.800620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.800556 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:32:12.802483 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.802453 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:32:12.802483 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.802464 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:32:12.802747 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.802727 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-kube-api-access-rwxvf" (OuterVolumeSpecName: "kube-api-access-rwxvf") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "kube-api-access-rwxvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:32:12.802747 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.802734 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a0ef8e10-5815-4175-ab7c-da4f50fe1116" (UID: "a0ef8e10-5815-4175-ab7c-da4f50fe1116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:32:12.902065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.902022 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwxvf\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-kube-api-access-rwxvf\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.902065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.902060 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.902065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.902070 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.902359 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.902079 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.902359 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.902087 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ef8e10-5815-4175-ab7c-da4f50fe1116-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.902359 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.902095 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:12.902359 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:12.902102 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ef8e10-5815-4175-ab7c-da4f50fe1116-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:32:13.650548 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:13.650516 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" Sep 08 16:32:13.651066 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:13.650557 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62" event={"ID":"a0ef8e10-5815-4175-ab7c-da4f50fe1116","Type":"ContainerDied","Data":"ceed8a62c9b2b946a45b7bf45aa94045ee0b876fa086047e43ac6ab62d62a11a"} Sep 08 16:32:13.651066 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:13.650616 2555 scope.go:117] "RemoveContainer" containerID="af1eac65d346024f9d1bbe6f7617804d476089e5894e535cf47a08818a0c2e19" Sep 08 16:32:13.671931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:13.671894 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62"] Sep 08 16:32:13.682324 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:13.677940 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1b78b587-openshift-default-5f6ccd674d-2fd62"] Sep 08 16:32:14.937930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:32:14.937878 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" path="/var/lib/kubelet/pods/a0ef8e10-5815-4175-ab7c-da4f50fe1116/volumes" Sep 08 16:33:34.832663 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:34.832624 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf"] Sep 08 16:33:45.734821 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:45.734780 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b"] Sep 08 16:33:45.735365 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:45.735039 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" podUID="1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" containerName="maas-api" containerID="cri-o://74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60" gracePeriod=30 Sep 08 16:33:45.984609 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:45.984583 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:33:46.100976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.100932 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-metrics-certs\") pod \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " Sep 08 16:33:46.101182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.101039 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k792z\" (UniqueName: \"kubernetes.io/projected/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-kube-api-access-k792z\") pod \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " Sep 08 16:33:46.101182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.101069 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls\") pod \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " Sep 08 16:33:46.101182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.101129 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-openshift-service-ca\") pod \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\" (UID: \"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0\") " Sep 08 16:33:46.101578 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.101531 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" (UID: "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:33:46.103192 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.103161 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" (UID: "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:33:46.103382 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.103190 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-kube-api-access-k792z" (OuterVolumeSpecName: "kube-api-access-k792z") pod "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" (UID: "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0"). InnerVolumeSpecName "kube-api-access-k792z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:33:46.103382 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.103203 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" (UID: "1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:33:46.200941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.200905 2555 generic.go:356] "Generic (PLEG): container finished" podID="1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" containerID="74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60" exitCode=0 Sep 08 16:33:46.201141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.200970 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" Sep 08 16:33:46.201141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.200981 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" event={"ID":"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0","Type":"ContainerDied","Data":"74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60"} Sep 08 16:33:46.201141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.201038 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b" event={"ID":"1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0","Type":"ContainerDied","Data":"94369d53d97440ccef7ac6be874cf02d72e8f8b8e191a6f41ee1b87dc3a1d10b"} Sep 08 16:33:46.201141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.201067 2555 scope.go:117] "RemoveContainer" containerID="74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60" Sep 08 16:33:46.202002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.201975 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:46.202126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.202004 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k792z\" (UniqueName: \"kubernetes.io/projected/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-kube-api-access-k792z\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:46.202126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.202019 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:46.202126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.202034 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:46.211773 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.211751 2555 scope.go:117] "RemoveContainer" containerID="74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60" Sep 08 16:33:46.212164 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:33:46.212141 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60\": container with ID starting with 74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60 not found: ID does not exist" containerID="74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60" Sep 08 16:33:46.212213 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.212174 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60"} err="failed to get container status \"74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60\": rpc error: code = NotFound desc = could not find container \"74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60\": container with ID starting with 74362eaa3c26a554b88c7be1e8e6516fed8a776613b74d09cc24fae4d33d0e60 not found: ID does not exist" Sep 08 16:33:46.251114 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.251076 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b"] Sep 08 16:33:46.262746 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.262715 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-293166-85c4865bdf-jth2b"] Sep 08 16:33:46.938478 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:46.938443 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" path="/var/lib/kubelet/pods/1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0/volumes" Sep 08 16:33:49.853288 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:49.853206 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" podUID="d2d64360-4feb-4a31-9e65-b2493514a2de" containerName="main" containerID="cri-o://57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a" gracePeriod=45 Sep 08 16:33:50.101917 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.101889 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:33:50.141817 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.141719 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d64360-4feb-4a31-9e65-b2493514a2de-tls-certs\") pod \"d2d64360-4feb-4a31-9e65-b2493514a2de\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " Sep 08 16:33:50.141995 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.141829 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-tmp-dir\") pod \"d2d64360-4feb-4a31-9e65-b2493514a2de\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " Sep 08 16:33:50.141995 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.141882 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd2wn\" (UniqueName: \"kubernetes.io/projected/d2d64360-4feb-4a31-9e65-b2493514a2de-kube-api-access-qd2wn\") pod \"d2d64360-4feb-4a31-9e65-b2493514a2de\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " Sep 08 16:33:50.141995 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.141914 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-model-cache\") pod \"d2d64360-4feb-4a31-9e65-b2493514a2de\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " Sep 08 16:33:50.141995 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.141960 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-dshm\") pod \"d2d64360-4feb-4a31-9e65-b2493514a2de\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " Sep 08 16:33:50.141995 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.141989 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-home\") pod \"d2d64360-4feb-4a31-9e65-b2493514a2de\" (UID: \"d2d64360-4feb-4a31-9e65-b2493514a2de\") " Sep 08 16:33:50.142237 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.142104 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2d64360-4feb-4a31-9e65-b2493514a2de" (UID: "d2d64360-4feb-4a31-9e65-b2493514a2de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:50.142357 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.142332 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-home" (OuterVolumeSpecName: "home") pod "d2d64360-4feb-4a31-9e65-b2493514a2de" (UID: "d2d64360-4feb-4a31-9e65-b2493514a2de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:50.142420 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.142398 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-tmp-dir\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:50.142482 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.142439 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2d64360-4feb-4a31-9e65-b2493514a2de" (UID: "d2d64360-4feb-4a31-9e65-b2493514a2de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:50.144180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.144147 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d64360-4feb-4a31-9e65-b2493514a2de-kube-api-access-qd2wn" (OuterVolumeSpecName: "kube-api-access-qd2wn") pod "d2d64360-4feb-4a31-9e65-b2493514a2de" (UID: "d2d64360-4feb-4a31-9e65-b2493514a2de"). InnerVolumeSpecName "kube-api-access-qd2wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:33:50.144331 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.144176 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d64360-4feb-4a31-9e65-b2493514a2de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2d64360-4feb-4a31-9e65-b2493514a2de" (UID: "d2d64360-4feb-4a31-9e65-b2493514a2de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:33:50.144331 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.144237 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-dshm" (OuterVolumeSpecName: "dshm") pod "d2d64360-4feb-4a31-9e65-b2493514a2de" (UID: "d2d64360-4feb-4a31-9e65-b2493514a2de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:50.228390 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.228351 2555 generic.go:356] "Generic (PLEG): container finished" podID="d2d64360-4feb-4a31-9e65-b2493514a2de" containerID="57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a" exitCode=0 Sep 08 16:33:50.228594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.228401 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" Sep 08 16:33:50.228594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.228431 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" event={"ID":"d2d64360-4feb-4a31-9e65-b2493514a2de","Type":"ContainerDied","Data":"57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a"} Sep 08 16:33:50.228594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.228475 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf" event={"ID":"d2d64360-4feb-4a31-9e65-b2493514a2de","Type":"ContainerDied","Data":"7c3aa35a4605ba6935e40801175734ccf10df96dc9c57f2b9788ccc9ae3d491a"} Sep 08 16:33:50.228594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.228495 2555 scope.go:117] "RemoveContainer" containerID="57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a" Sep 08 16:33:50.239196 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.239175 2555 scope.go:117] "RemoveContainer" containerID="57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a" Sep 08 16:33:50.239567 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:33:50.239545 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a\": container with ID starting with 57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a not found: ID does not exist" containerID="57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a" Sep 08 16:33:50.239636 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.239575 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a"} err="failed to get container status \"57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a\": rpc error: code = NotFound desc = could not find container \"57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a\": container with ID starting with 57cf6f680267b9f54e431d964b846325ebe700f33a16f2af93333d923b479a8a not found: ID does not exist" Sep 08 16:33:50.243878 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.243853 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2wn\" (UniqueName: \"kubernetes.io/projected/d2d64360-4feb-4a31-9e65-b2493514a2de-kube-api-access-qd2wn\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:50.243878 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.243879 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-model-cache\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:50.244005 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.243888 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-dshm\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:50.244005 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.243896 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2d64360-4feb-4a31-9e65-b2493514a2de-home\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:50.244005 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.243903 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d64360-4feb-4a31-9e65-b2493514a2de-tls-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:50.251997 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.251972 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf"] Sep 08 16:33:50.258434 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.258403 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-293166/autores-model-293166-kserve-c945478d7-jfvkf"] Sep 08 16:33:50.936770 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:50.936733 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d64360-4feb-4a31-9e65-b2493514a2de" path="/var/lib/kubelet/pods/d2d64360-4feb-4a31-9e65-b2493514a2de/volumes" Sep 08 16:33:53.103917 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:53.103885 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b"] Sep 08 16:33:53.104372 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:53.104117 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" containerID="cri-o://ba6791c5e71a9dcccab790b7cd24013eb8fa7294772f83163dbc51115f63adc2" gracePeriod=30 Sep 08 16:33:58.278625 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.278592 2555 generic.go:356] "Generic (PLEG): container finished" podID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerID="ba6791c5e71a9dcccab790b7cd24013eb8fa7294772f83163dbc51115f63adc2" exitCode=0 Sep 08 16:33:58.278976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.278662 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" event={"ID":"b4076516-c2ce-4d4c-b6de-78c94c179e41","Type":"ContainerDied","Data":"ba6791c5e71a9dcccab790b7cd24013eb8fa7294772f83163dbc51115f63adc2"} Sep 08 16:33:58.278976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.278728 2555 scope.go:117] "RemoveContainer" containerID="405fda300f9af6898e867a779323810cdf4fa71d14479f6ea8f4179375e091f8" Sep 08 16:33:58.359262 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.359220 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:33:58.419734 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.419645 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-data\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.419734 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.419684 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-envoy\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.419975 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.419747 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdg27\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-kube-api-access-cdg27\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.419975 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.419767 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-credential-socket\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.419975 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.419803 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-socket\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.420142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.419981 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:58.420142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420007 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4076516-c2ce-4d4c-b6de-78c94c179e41-istiod-ca-cert\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.420142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420074 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-certs\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.420142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420097 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:58.420142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420118 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-podinfo\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.420142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420108 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:58.420471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420145 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-token\") pod \"b4076516-c2ce-4d4c-b6de-78c94c179e41\" (UID: \"b4076516-c2ce-4d4c-b6de-78c94c179e41\") " Sep 08 16:33:58.420471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420304 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:58.420471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420344 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4076516-c2ce-4d4c-b6de-78c94c179e41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:33:58.420634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420537 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.420634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420550 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.420634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420559 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4076516-c2ce-4d4c-b6de-78c94c179e41-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.420634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420568 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.420634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.420575 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.422209 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.422153 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:33:58.422971 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.422946 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-kube-api-access-cdg27" (OuterVolumeSpecName: "kube-api-access-cdg27") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "kube-api-access-cdg27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:33:58.423085 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.422981 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:33:58.423163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.423136 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4076516-c2ce-4d4c-b6de-78c94c179e41" (UID: "b4076516-c2ce-4d4c-b6de-78c94c179e41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:33:58.504147 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504104 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr"] Sep 08 16:33:58.504755 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504736 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" Sep 08 16:33:58.504808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504757 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" Sep 08 16:33:58.504808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504767 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" Sep 08 16:33:58.504808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504774 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" Sep 08 16:33:58.504808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504788 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d64360-4feb-4a31-9e65-b2493514a2de" containerName="main" Sep 08 16:33:58.504808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504797 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d64360-4feb-4a31-9e65-b2493514a2de" containerName="main" Sep 08 16:33:58.504965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504813 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" containerName="maas-api" Sep 08 16:33:58.504965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504822 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" containerName="maas-api" Sep 08 16:33:58.504965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504837 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" Sep 08 16:33:58.504965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504845 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" Sep 08 16:33:58.504965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504953 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" Sep 08 16:33:58.504965 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504966 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" Sep 08 16:33:58.505138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504979 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d64360-4feb-4a31-9e65-b2493514a2de" containerName="main" Sep 08 16:33:58.505138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.504986 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a5c52d1-ee4f-404f-b88e-b5515ebcbfd0" containerName="maas-api" Sep 08 16:33:58.507292 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.507273 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.511496 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.511471 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ebb057-serving-cert\"" Sep 08 16:33:58.516933 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.516910 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr"] Sep 08 16:33:58.521109 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.521083 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.521109 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.521103 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.521109 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.521114 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4076516-c2ce-4d4c-b6de-78c94c179e41-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.521410 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.521122 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdg27\" (UniqueName: \"kubernetes.io/projected/b4076516-c2ce-4d4c-b6de-78c94c179e41-kube-api-access-cdg27\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:33:58.621710 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.621667 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66qc\" (UniqueName: \"kubernetes.io/projected/f6c95eaa-9fd2-427c-a873-5c9fb2994268-kube-api-access-x66qc\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.621710 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.621713 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c95eaa-9fd2-427c-a873-5c9fb2994268-openshift-service-ca\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.621964 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.621755 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-metrics-certs\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.621964 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.621837 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-maas-api-tls\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.722630 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.722531 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-maas-api-tls\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.722805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.722635 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66qc\" (UniqueName: \"kubernetes.io/projected/f6c95eaa-9fd2-427c-a873-5c9fb2994268-kube-api-access-x66qc\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.722805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.722656 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c95eaa-9fd2-427c-a873-5c9fb2994268-openshift-service-ca\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.722805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.722680 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-metrics-certs\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.723465 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.723441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c95eaa-9fd2-427c-a873-5c9fb2994268-openshift-service-ca\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.724951 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.724928 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-metrics-certs\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.725002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.724977 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-maas-api-tls\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.730779 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.730744 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66qc\" (UniqueName: \"kubernetes.io/projected/f6c95eaa-9fd2-427c-a873-5c9fb2994268-kube-api-access-x66qc\") pod \"maas-api-e2e-explicit-ebb057-77775b466b-4hdlr\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:58.819064 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:58.819018 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:33:59.098714 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:59.098686 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr"] Sep 08 16:33:59.100522 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:33:59.100486 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c95eaa_9fd2_427c_a873_5c9fb2994268.slice/crio-cccf57a0253268754e6f4e455a52d70f7d16bbcb2bfd7ad098841d8de5a3b27b WatchSource:0}: Error finding container cccf57a0253268754e6f4e455a52d70f7d16bbcb2bfd7ad098841d8de5a3b27b: Status 404 returned error can't find the container with id cccf57a0253268754e6f4e455a52d70f7d16bbcb2bfd7ad098841d8de5a3b27b Sep 08 16:33:59.285933 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:59.285903 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" Sep 08 16:33:59.286453 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:59.285931 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b" event={"ID":"b4076516-c2ce-4d4c-b6de-78c94c179e41","Type":"ContainerDied","Data":"33b47b140ff707f713c65b1d2d30be0b7a671bebd548b59fa2443f61a4891b35"} Sep 08 16:33:59.286453 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:59.286000 2555 scope.go:117] "RemoveContainer" containerID="ba6791c5e71a9dcccab790b7cd24013eb8fa7294772f83163dbc51115f63adc2" Sep 08 16:33:59.287555 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:59.287369 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" event={"ID":"f6c95eaa-9fd2-427c-a873-5c9fb2994268","Type":"ContainerStarted","Data":"cccf57a0253268754e6f4e455a52d70f7d16bbcb2bfd7ad098841d8de5a3b27b"} Sep 08 16:33:59.312616 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:59.312582 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b"] Sep 08 16:33:59.326358 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:33:59.326325 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-293166-openshift-default-599d8647fb-gp64b"] Sep 08 16:34:00.295640 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:00.295604 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" event={"ID":"f6c95eaa-9fd2-427c-a873-5c9fb2994268","Type":"ContainerStarted","Data":"1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88"} Sep 08 16:34:00.296089 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:00.295671 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:34:00.317407 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:00.317348 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" podStartSLOduration=1.682003514 podStartE2EDuration="2.31732967s" totalImagesPullingTime="635.326156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:00.316612533 +0000 UTC m=+1273.911611313" watchObservedRunningTime="2026-09-08 16:34:00.31732967 +0000 UTC m=+1273.912328428" Sep 08 16:34:00.937386 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:00.937348 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" path="/var/lib/kubelet/pods/b4076516-c2ce-4d4c-b6de-78c94c179e41/volumes" Sep 08 16:34:06.308522 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:06.308490 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:34:26.861014 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.860972 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg"] Sep 08 16:34:26.861615 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.861593 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" Sep 08 16:34:26.861700 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.861619 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" Sep 08 16:34:26.861756 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.861718 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ef8e10-5815-4175-ab7c-da4f50fe1116" containerName="istio-proxy" Sep 08 16:34:26.861756 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.861733 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4076516-c2ce-4d4c-b6de-78c94c179e41" containerName="istio-proxy" Sep 08 16:34:26.864952 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.864927 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.867571 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.867551 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e22de8-openshift-default-dockercfg-vwbjg\"" Sep 08 16:34:26.878286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.878257 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg"] Sep 08 16:34:26.894547 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894510 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-podinfo\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894707 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894555 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c25e40-7029-4c25-9e4d-4508b1363ce5-istiod-ca-cert\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894707 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894636 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-socket\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894707 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894686 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-certs\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894714 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278pq\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-kube-api-access-278pq\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894741 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-envoy\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894791 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-credential-socket\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894822 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-token\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.894956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.894874 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-data\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.995735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.995688 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278pq\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-kube-api-access-278pq\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.995923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.995731 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-envoy\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.995923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.995825 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-credential-socket\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.995923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.995880 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-token\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996084 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996005 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-data\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996084 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996073 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-podinfo\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996190 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996119 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c25e40-7029-4c25-9e4d-4508b1363ce5-istiod-ca-cert\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996282 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996193 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-socket\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996357 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996276 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-certs\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996357 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996346 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-credential-socket\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996481 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-data\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996498 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-socket\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.996687 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.996577 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-certs\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.997293 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.997267 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c25e40-7029-4c25-9e4d-4508b1363ce5-istiod-ca-cert\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.998238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.998214 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-envoy\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:26.998601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:26.998580 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-podinfo\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:27.004673 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.004650 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-token\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:27.004886 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.004864 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-278pq\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-kube-api-access-278pq\") pod \"e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:27.175197 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.175101 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:27.460595 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.460568 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg"] Sep 08 16:34:27.462431 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:34:27.462400 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c25e40_7029_4c25_9e4d_4508b1363ce5.slice/crio-46d4c074480b8710e5d6d5e9840b5f47ab53dd56afd07965441c3d26d7a59d4a WatchSource:0}: Error finding container 46d4c074480b8710e5d6d5e9840b5f47ab53dd56afd07965441c3d26d7a59d4a: Status 404 returned error can't find the container with id 46d4c074480b8710e5d6d5e9840b5f47ab53dd56afd07965441c3d26d7a59d4a Sep 08 16:34:27.464737 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.464701 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:27.464802 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.464768 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:27.464802 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.464794 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:27.573174 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.573136 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74"] Sep 08 16:34:27.577476 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.577449 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.580219 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.580192 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ebb057\"/\"default-dockercfg-9p6cv\"" Sep 08 16:34:27.580474 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.580442 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ebb057\"/\"explicit-model-ebb057-kserve-self-signed-certs\"" Sep 08 16:34:27.580650 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.580454 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ebb057\"/\"kube-root-ca.crt\"" Sep 08 16:34:27.580770 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.580542 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ebb057\"/\"openshift-service-ca.crt\"" Sep 08 16:34:27.591435 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.591402 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74"] Sep 08 16:34:27.701920 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.701877 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b11f73-d4d2-4789-8128-8080f5848902-tls-certs\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.702092 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.701927 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-home\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.702092 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.702051 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjb9h\" (UniqueName: \"kubernetes.io/projected/79b11f73-d4d2-4789-8128-8080f5848902-kube-api-access-fjb9h\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.702169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.702107 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-model-cache\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.702169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.702142 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-tmp-dir\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.702239 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.702173 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-dshm\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.802946 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.802859 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b11f73-d4d2-4789-8128-8080f5848902-tls-certs\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.802946 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.802907 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-home\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.803148 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.802969 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjb9h\" (UniqueName: \"kubernetes.io/projected/79b11f73-d4d2-4789-8128-8080f5848902-kube-api-access-fjb9h\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.803148 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.802995 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-model-cache\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.803148 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.803016 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-tmp-dir\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.803148 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.803136 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-dshm\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.803459 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.803435 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-model-cache\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.803540 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.803477 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-home\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.803540 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.803441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-tmp-dir\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.805280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.805239 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-dshm\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.805458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.805441 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b11f73-d4d2-4789-8128-8080f5848902-tls-certs\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.814163 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.814139 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjb9h\" (UniqueName: \"kubernetes.io/projected/79b11f73-d4d2-4789-8128-8080f5848902-kube-api-access-fjb9h\") pod \"explicit-model-ebb057-kserve-d7fcd664c-dkr74\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:27.887883 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:27.887844 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:28.162534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:28.162507 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74"] Sep 08 16:34:28.164319 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:34:28.164288 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79b11f73_d4d2_4789_8128_8080f5848902.slice/crio-24cbb03f87134ed9aadab79ea7d1360a8823fb9138299f9948d32bbeacb330b9 WatchSource:0}: Error finding container 24cbb03f87134ed9aadab79ea7d1360a8823fb9138299f9948d32bbeacb330b9: Status 404 returned error can't find the container with id 24cbb03f87134ed9aadab79ea7d1360a8823fb9138299f9948d32bbeacb330b9 Sep 08 16:34:28.455677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:28.455568 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" event={"ID":"50c25e40-7029-4c25-9e4d-4508b1363ce5","Type":"ContainerStarted","Data":"1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c"} Sep 08 16:34:28.455677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:28.455636 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" event={"ID":"50c25e40-7029-4c25-9e4d-4508b1363ce5","Type":"ContainerStarted","Data":"46d4c074480b8710e5d6d5e9840b5f47ab53dd56afd07965441c3d26d7a59d4a"} Sep 08 16:34:28.457162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:28.457133 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" event={"ID":"79b11f73-d4d2-4789-8128-8080f5848902","Type":"ContainerStarted","Data":"a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f"} Sep 08 16:34:28.457332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:28.457169 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" event={"ID":"79b11f73-d4d2-4789-8128-8080f5848902","Type":"ContainerStarted","Data":"24cbb03f87134ed9aadab79ea7d1360a8823fb9138299f9948d32bbeacb330b9"} Sep 08 16:34:28.479306 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:28.479227 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" podStartSLOduration=2.4792099690000002 podStartE2EDuration="2.479209969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:28.476332807 +0000 UTC m=+1302.071331591" watchObservedRunningTime="2026-09-08 16:34:28.479209969 +0000 UTC m=+1302.074208728" Sep 08 16:34:28.512609 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:28.512554 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" podStartSLOduration=1.512537167 podStartE2EDuration="1.512537167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:28.511172057 +0000 UTC m=+1302.106170835" watchObservedRunningTime="2026-09-08 16:34:28.512537167 +0000 UTC m=+1302.107535926" Sep 08 16:34:29.176273 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:29.176203 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:29.181530 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:29.181504 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:29.463781 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:29.463687 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:29.464555 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:29.464533 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:30.279501 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:30.279464 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74"] Sep 08 16:34:37.888106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:37.888058 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:41.477659 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.477625 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr"] Sep 08 16:34:41.478153 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.477840 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" podUID="f6c95eaa-9fd2-427c-a873-5c9fb2994268" containerName="maas-api" containerID="cri-o://1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88" gracePeriod=30 Sep 08 16:34:41.783056 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.782962 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:34:41.960271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.959597 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c95eaa-9fd2-427c-a873-5c9fb2994268-openshift-service-ca\") pod \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " Sep 08 16:34:41.960271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.959744 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x66qc\" (UniqueName: \"kubernetes.io/projected/f6c95eaa-9fd2-427c-a873-5c9fb2994268-kube-api-access-x66qc\") pod \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " Sep 08 16:34:41.960271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.959803 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-maas-api-tls\") pod \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " Sep 08 16:34:41.960271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.959832 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-metrics-certs\") pod \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\" (UID: \"f6c95eaa-9fd2-427c-a873-5c9fb2994268\") " Sep 08 16:34:41.961016 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.960983 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c95eaa-9fd2-427c-a873-5c9fb2994268-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6c95eaa-9fd2-427c-a873-5c9fb2994268" (UID: "f6c95eaa-9fd2-427c-a873-5c9fb2994268"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:34:41.964532 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.964375 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6c95eaa-9fd2-427c-a873-5c9fb2994268" (UID: "f6c95eaa-9fd2-427c-a873-5c9fb2994268"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:34:41.964532 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.964487 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6c95eaa-9fd2-427c-a873-5c9fb2994268" (UID: "f6c95eaa-9fd2-427c-a873-5c9fb2994268"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:34:41.965395 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:41.965359 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c95eaa-9fd2-427c-a873-5c9fb2994268-kube-api-access-x66qc" (OuterVolumeSpecName: "kube-api-access-x66qc") pod "f6c95eaa-9fd2-427c-a873-5c9fb2994268" (UID: "f6c95eaa-9fd2-427c-a873-5c9fb2994268"). InnerVolumeSpecName "kube-api-access-x66qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:34:42.061300 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.061165 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66qc\" (UniqueName: \"kubernetes.io/projected/f6c95eaa-9fd2-427c-a873-5c9fb2994268-kube-api-access-x66qc\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:42.061300 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.061219 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:42.061300 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.061235 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c95eaa-9fd2-427c-a873-5c9fb2994268-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:42.061300 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.061264 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c95eaa-9fd2-427c-a873-5c9fb2994268-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:42.580115 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.580077 2555 generic.go:356] "Generic (PLEG): container finished" podID="f6c95eaa-9fd2-427c-a873-5c9fb2994268" containerID="1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88" exitCode=0 Sep 08 16:34:42.580601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.580127 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" Sep 08 16:34:42.580601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.580138 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" event={"ID":"f6c95eaa-9fd2-427c-a873-5c9fb2994268","Type":"ContainerDied","Data":"1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88"} Sep 08 16:34:42.580601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.580180 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr" event={"ID":"f6c95eaa-9fd2-427c-a873-5c9fb2994268","Type":"ContainerDied","Data":"cccf57a0253268754e6f4e455a52d70f7d16bbcb2bfd7ad098841d8de5a3b27b"} Sep 08 16:34:42.580601 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.580201 2555 scope.go:117] "RemoveContainer" containerID="1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88" Sep 08 16:34:42.590500 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.590481 2555 scope.go:117] "RemoveContainer" containerID="1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88" Sep 08 16:34:42.590818 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:34:42.590795 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88\": container with ID starting with 1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88 not found: ID does not exist" containerID="1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88" Sep 08 16:34:42.590895 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.590827 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88"} err="failed to get container status \"1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88\": rpc error: code = NotFound desc = could not find container \"1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88\": container with ID starting with 1fa05b49f2e1b073557d783f461d286d38a282ee97daec04b85914c0332c2a88 not found: ID does not exist" Sep 08 16:34:42.605705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.605674 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr"] Sep 08 16:34:42.612307 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.612279 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ebb057-77775b466b-4hdlr"] Sep 08 16:34:42.937125 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:42.937086 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c95eaa-9fd2-427c-a873-5c9fb2994268" path="/var/lib/kubelet/pods/f6c95eaa-9fd2-427c-a873-5c9fb2994268/volumes" Sep 08 16:34:45.490733 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.490684 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" podUID="79b11f73-d4d2-4789-8128-8080f5848902" containerName="main" containerID="cri-o://a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f" gracePeriod=45 Sep 08 16:34:45.755568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.749671 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:45.900195 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900155 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-home\") pod \"79b11f73-d4d2-4789-8128-8080f5848902\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " Sep 08 16:34:45.900382 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900286 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjb9h\" (UniqueName: \"kubernetes.io/projected/79b11f73-d4d2-4789-8128-8080f5848902-kube-api-access-fjb9h\") pod \"79b11f73-d4d2-4789-8128-8080f5848902\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " Sep 08 16:34:45.900382 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900330 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-tmp-dir\") pod \"79b11f73-d4d2-4789-8128-8080f5848902\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " Sep 08 16:34:45.900473 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900384 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b11f73-d4d2-4789-8128-8080f5848902-tls-certs\") pod \"79b11f73-d4d2-4789-8128-8080f5848902\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " Sep 08 16:34:45.900473 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900422 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-dshm\") pod \"79b11f73-d4d2-4789-8128-8080f5848902\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " Sep 08 16:34:45.900473 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900454 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-model-cache\") pod \"79b11f73-d4d2-4789-8128-8080f5848902\" (UID: \"79b11f73-d4d2-4789-8128-8080f5848902\") " Sep 08 16:34:45.900473 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900456 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-home" (OuterVolumeSpecName: "home") pod "79b11f73-d4d2-4789-8128-8080f5848902" (UID: "79b11f73-d4d2-4789-8128-8080f5848902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:34:45.900683 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900641 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79b11f73-d4d2-4789-8128-8080f5848902" (UID: "79b11f73-d4d2-4789-8128-8080f5848902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:34:45.900780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900754 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-model-cache" (OuterVolumeSpecName: "model-cache") pod "79b11f73-d4d2-4789-8128-8080f5848902" (UID: "79b11f73-d4d2-4789-8128-8080f5848902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:34:45.900883 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900858 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-tmp-dir\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:45.900994 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.900885 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-home\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:45.902696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.902670 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b11f73-d4d2-4789-8128-8080f5848902-kube-api-access-fjb9h" (OuterVolumeSpecName: "kube-api-access-fjb9h") pod "79b11f73-d4d2-4789-8128-8080f5848902" (UID: "79b11f73-d4d2-4789-8128-8080f5848902"). InnerVolumeSpecName "kube-api-access-fjb9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:34:45.902812 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.902706 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b11f73-d4d2-4789-8128-8080f5848902-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79b11f73-d4d2-4789-8128-8080f5848902" (UID: "79b11f73-d4d2-4789-8128-8080f5848902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:34:45.902967 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:45.902941 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-dshm" (OuterVolumeSpecName: "dshm") pod "79b11f73-d4d2-4789-8128-8080f5848902" (UID: "79b11f73-d4d2-4789-8128-8080f5848902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:34:46.001423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.001381 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjb9h\" (UniqueName: \"kubernetes.io/projected/79b11f73-d4d2-4789-8128-8080f5848902-kube-api-access-fjb9h\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:46.001423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.001416 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b11f73-d4d2-4789-8128-8080f5848902-tls-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:46.001423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.001427 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-dshm\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:46.001423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.001434 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b11f73-d4d2-4789-8128-8080f5848902-model-cache\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:34:46.605566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.605522 2555 generic.go:356] "Generic (PLEG): container finished" podID="79b11f73-d4d2-4789-8128-8080f5848902" containerID="a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f" exitCode=0 Sep 08 16:34:46.606035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.605575 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" Sep 08 16:34:46.606035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.605610 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" event={"ID":"79b11f73-d4d2-4789-8128-8080f5848902","Type":"ContainerDied","Data":"a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f"} Sep 08 16:34:46.606035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.605658 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74" event={"ID":"79b11f73-d4d2-4789-8128-8080f5848902","Type":"ContainerDied","Data":"24cbb03f87134ed9aadab79ea7d1360a8823fb9138299f9948d32bbeacb330b9"} Sep 08 16:34:46.606035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.605680 2555 scope.go:117] "RemoveContainer" containerID="a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f" Sep 08 16:34:46.616237 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.616215 2555 scope.go:117] "RemoveContainer" containerID="a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f" Sep 08 16:34:46.616661 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:34:46.616639 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f\": container with ID starting with a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f not found: ID does not exist" containerID="a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f" Sep 08 16:34:46.616736 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.616674 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f"} err="failed to get container status \"a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f\": rpc error: code = NotFound desc = could not find container \"a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f\": container with ID starting with a0a2ec3e93ea089c1c4c8c8f6629a41d540dad358334880a8b5024dedc21799f not found: ID does not exist" Sep 08 16:34:46.631681 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.631640 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74"] Sep 08 16:34:46.636732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.636693 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ebb057/explicit-model-ebb057-kserve-d7fcd664c-dkr74"] Sep 08 16:34:46.939528 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:46.939449 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b11f73-d4d2-4789-8128-8080f5848902" path="/var/lib/kubelet/pods/79b11f73-d4d2-4789-8128-8080f5848902/volumes" Sep 08 16:34:49.625523 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:49.625490 2555 generic.go:356] "Generic (PLEG): container finished" podID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerID="1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c" exitCode=0 Sep 08 16:34:49.625978 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:49.625572 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" event={"ID":"50c25e40-7029-4c25-9e4d-4508b1363ce5","Type":"ContainerDied","Data":"1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c"} Sep 08 16:34:49.626037 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:49.626009 2555 scope.go:117] "RemoveContainer" containerID="1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c" Sep 08 16:34:49.627110 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:49.627073 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:49.627173 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:49.627153 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:49.627213 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:49.627184 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:50.583099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.583062 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9"] Sep 08 16:34:50.584216 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.584186 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c95eaa-9fd2-427c-a873-5c9fb2994268" containerName="maas-api" Sep 08 16:34:50.584216 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.584216 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c95eaa-9fd2-427c-a873-5c9fb2994268" containerName="maas-api" Sep 08 16:34:50.584407 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.584232 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b11f73-d4d2-4789-8128-8080f5848902" containerName="main" Sep 08 16:34:50.584407 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.584238 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b11f73-d4d2-4789-8128-8080f5848902" containerName="main" Sep 08 16:34:50.584480 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.584421 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b11f73-d4d2-4789-8128-8080f5848902" containerName="main" Sep 08 16:34:50.584480 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.584437 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c95eaa-9fd2-427c-a873-5c9fb2994268" containerName="maas-api" Sep 08 16:34:50.586755 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.586729 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.589888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.589863 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-7a2cf0-openshift-default-dockercfg-847mz\"" Sep 08 16:34:50.598342 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.598310 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9"] Sep 08 16:34:50.633007 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.632966 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" event={"ID":"50c25e40-7029-4c25-9e4d-4508b1363ce5","Type":"ContainerStarted","Data":"34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46"} Sep 08 16:34:50.639468 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639439 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-envoy\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639491 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-certs\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639545 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-token\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639577 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istiod-ca-cert\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639601 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bb6x\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-kube-api-access-2bb6x\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639655 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-credential-socket\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639757 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-socket\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639818 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-data\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.639912 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.639885 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-podinfo\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.740805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.740753 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istiod-ca-cert\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.740999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.740821 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bb6x\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-kube-api-access-2bb6x\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.740999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.740875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-credential-socket\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.740999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.740932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-socket\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.740999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.740955 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-data\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.741238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.741014 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-podinfo\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.741238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.741071 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-envoy\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.741238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.741116 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-certs\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.741238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.741148 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-token\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.742353 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.742316 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istiod-ca-cert\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.742957 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.742927 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-credential-socket\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.743207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.743184 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-socket\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.743478 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.743455 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-data\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.744424 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.744388 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-certs\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.754285 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.746680 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-podinfo\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.754285 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.748964 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-envoy\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.754285 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.751590 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-token\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.769488 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.769445 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bb6x\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-kube-api-access-2bb6x\") pod \"e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:50.900586 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:50.900544 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:51.176363 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.176232 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:51.176620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.176594 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 08 16:34:51.176685 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.176665 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 08 16:34:51.224405 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.224377 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9"] Sep 08 16:34:51.225849 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:34:51.225820 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df56df2_7e2e_436e_b4c3_cccd1c8fdc7f.slice/crio-c6adddf4f6567466ec17e0810914bbb5e03e0d9d41dc3d1b7916258dfdceaaf4 WatchSource:0}: Error finding container c6adddf4f6567466ec17e0810914bbb5e03e0d9d41dc3d1b7916258dfdceaaf4: Status 404 returned error can't find the container with id c6adddf4f6567466ec17e0810914bbb5e03e0d9d41dc3d1b7916258dfdceaaf4 Sep 08 16:34:51.228060 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.228026 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:51.228126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.228096 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:51.228162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.228131 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:34:51.643325 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.643160 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" event={"ID":"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f","Type":"ContainerStarted","Data":"2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d"} Sep 08 16:34:51.643325 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.643222 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" event={"ID":"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f","Type":"ContainerStarted","Data":"c6adddf4f6567466ec17e0810914bbb5e03e0d9d41dc3d1b7916258dfdceaaf4"} Sep 08 16:34:51.679737 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.679658 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" podStartSLOduration=1.679639818 podStartE2EDuration="1.679639818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:51.676882751 +0000 UTC m=+1325.271881530" watchObservedRunningTime="2026-09-08 16:34:51.679639818 +0000 UTC m=+1325.274638576" Sep 08 16:34:51.901071 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:51.900975 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:52.176335 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:52.176207 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 08 16:34:52.176335 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:52.176306 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 08 16:34:52.905971 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:52.905942 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:53.176360 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:53.176232 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 08 16:34:53.176360 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:53.176326 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 08 16:34:53.661374 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:53.661338 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:53.666154 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:53.666118 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:34:54.176552 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:54.176507 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 08 16:34:54.177044 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:54.176573 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 08 16:34:55.175626 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:55.175589 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 08 16:34:55.175821 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:55.175646 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 08 16:34:56.180606 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:56.180569 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:56.181096 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:56.180971 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:34:56.181797 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:34:56.181780 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:35:09.273415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.273325 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl"] Sep 08 16:35:09.276550 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.276524 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.280058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.280022 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"openshift-service-ca.crt\"" Sep 08 16:35:09.280199 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.280063 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"kube-root-ca.crt\"" Sep 08 16:35:09.280199 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.280140 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"auth-test-model-bce20b-kserve-self-signed-certs\"" Sep 08 16:35:09.280431 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.280409 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bce20b\"/\"default-dockercfg-6sw9q\"" Sep 08 16:35:09.291421 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.291385 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl"] Sep 08 16:35:09.328691 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.328645 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f45bfb-a143-4ccf-a7e9-f618ef161560-tls-certs\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.328691 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.328693 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-dshm\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.328916 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.328713 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-tmp-dir\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.328916 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.328797 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-model-cache\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.328916 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.328845 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-home\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.328916 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.328868 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2f7\" (UniqueName: \"kubernetes.io/projected/77f45bfb-a143-4ccf-a7e9-f618ef161560-kube-api-access-4t2f7\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.429594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.429553 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f45bfb-a143-4ccf-a7e9-f618ef161560-tls-certs\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.429780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.429606 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-dshm\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.429780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.429719 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-tmp-dir\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.429874 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.429817 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-model-cache\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.429936 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.429902 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-home\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.429993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.429934 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2f7\" (UniqueName: \"kubernetes.io/projected/77f45bfb-a143-4ccf-a7e9-f618ef161560-kube-api-access-4t2f7\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.430125 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.430098 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-tmp-dir\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.430194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.430161 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-model-cache\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.430275 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.430218 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-home\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.432445 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.432424 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-dshm\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.432793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.432767 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f45bfb-a143-4ccf-a7e9-f618ef161560-tls-certs\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.438651 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.438620 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2f7\" (UniqueName: \"kubernetes.io/projected/77f45bfb-a143-4ccf-a7e9-f618ef161560-kube-api-access-4t2f7\") pod \"auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.587871 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.587769 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:09.865304 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:09.865275 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl"] Sep 08 16:35:09.866353 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:35:09.866322 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77f45bfb_a143_4ccf_a7e9_f618ef161560.slice/crio-69c5bebba846372bfa26333ece151dadb696760f57f41b422f96e3da4cfb95d5 WatchSource:0}: Error finding container 69c5bebba846372bfa26333ece151dadb696760f57f41b422f96e3da4cfb95d5: Status 404 returned error can't find the container with id 69c5bebba846372bfa26333ece151dadb696760f57f41b422f96e3da4cfb95d5 Sep 08 16:35:10.766588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:10.766546 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" event={"ID":"77f45bfb-a143-4ccf-a7e9-f618ef161560","Type":"ContainerStarted","Data":"caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642"} Sep 08 16:35:10.767180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:10.767141 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" event={"ID":"77f45bfb-a143-4ccf-a7e9-f618ef161560","Type":"ContainerStarted","Data":"69c5bebba846372bfa26333ece151dadb696760f57f41b422f96e3da4cfb95d5"} Sep 08 16:35:10.799222 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:10.792779 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" podStartSLOduration=1.7927577239999999 podStartE2EDuration="1.792757724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:10.787514667 +0000 UTC m=+1344.382513426" watchObservedRunningTime="2026-09-08 16:35:10.792757724 +0000 UTC m=+1344.387756485" Sep 08 16:35:10.920619 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:10.920583 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9"] Sep 08 16:35:10.920843 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:10.920818 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" podUID="3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" containerName="istio-proxy" containerID="cri-o://2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d" gracePeriod=30 Sep 08 16:35:16.184479 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.184414 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:35:16.296656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296598 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-token\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.296656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296664 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bb6x\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-kube-api-access-2bb6x\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.296891 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296747 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-socket\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.296891 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296771 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-data\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.296891 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296856 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-podinfo\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.297020 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296903 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-certs\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.297020 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296942 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-credential-socket\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.297020 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.296980 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istiod-ca-cert\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.297020 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.297012 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-envoy\") pod \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\" (UID: \"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f\") " Sep 08 16:35:16.299566 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.299533 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-data" (OuterVolumeSpecName: "istio-data") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:35:16.299683 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.299555 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:35:16.301282 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.301234 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-kube-api-access-2bb6x" (OuterVolumeSpecName: "kube-api-access-2bb6x") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "kube-api-access-2bb6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:35:16.301359 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.301341 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-token" (OuterVolumeSpecName: "istio-token") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:35:16.302431 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.302404 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:35:16.302557 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.302521 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:35:16.302627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.302595 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:35:16.303973 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.303954 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:35:16.304049 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.304030 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" (UID: "3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:35:16.398692 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398639 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.398692 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398685 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bb6x\" (UniqueName: \"kubernetes.io/projected/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-kube-api-access-2bb6x\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.398692 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398698 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.398692 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398707 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.398692 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398715 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.399010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398723 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.399010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398739 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.399010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398748 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.399010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.398756 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:16.804495 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.804456 2555 generic.go:356] "Generic (PLEG): container finished" podID="3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" containerID="2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d" exitCode=0 Sep 08 16:35:16.804696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.804501 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" Sep 08 16:35:16.804696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.804542 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" event={"ID":"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f","Type":"ContainerDied","Data":"2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d"} Sep 08 16:35:16.804696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.804596 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9" event={"ID":"3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f","Type":"ContainerDied","Data":"c6adddf4f6567466ec17e0810914bbb5e03e0d9d41dc3d1b7916258dfdceaaf4"} Sep 08 16:35:16.804696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.804627 2555 scope.go:117] "RemoveContainer" containerID="2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d" Sep 08 16:35:16.815196 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.815178 2555 scope.go:117] "RemoveContainer" containerID="2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d" Sep 08 16:35:16.815511 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:35:16.815488 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d\": container with ID starting with 2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d not found: ID does not exist" containerID="2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d" Sep 08 16:35:16.815579 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.815525 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d"} err="failed to get container status \"2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d\": rpc error: code = NotFound desc = could not find container \"2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d\": container with ID starting with 2e94adb73ffee54380a716d2b631d4950e971f38661af568c9e9b150f19c597d not found: ID does not exist" Sep 08 16:35:16.832136 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.832092 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9"] Sep 08 16:35:16.836763 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.836705 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-7a2cf0-openshift-default-59bdf9d494-fv6p9"] Sep 08 16:35:16.937352 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:16.937316 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" path="/var/lib/kubelet/pods/3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f/volumes" Sep 08 16:35:19.588180 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:19.588138 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:19.588636 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:19.588196 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:19.601305 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:19.601266 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:19.834413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:19.834381 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:35:23.403768 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.403731 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx"] Sep 08 16:35:23.404162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.404149 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" containerName="istio-proxy" Sep 08 16:35:23.404208 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.404164 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" containerName="istio-proxy" Sep 08 16:35:23.404259 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.404228 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df56df2-7e2e-436e-b4c3-cccd1c8fdc7f" containerName="istio-proxy" Sep 08 16:35:23.408892 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.408869 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.411506 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.411487 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-8ded33-openshift-default-dockercfg-5swsd\"" Sep 08 16:35:23.425577 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.425549 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx"] Sep 08 16:35:23.566984 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.566928 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-credential-socket\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567005 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-token\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567055 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-certs\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567076 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-podinfo\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567096 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-data\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567119 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-socket\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567142 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54af582b-54c5-40ac-85ac-d1bd199c00fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567174 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4nq8\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-kube-api-access-b4nq8\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.567205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.567195 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-envoy\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-certs\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668469 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-podinfo\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668493 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-data\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668516 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-socket\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668538 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54af582b-54c5-40ac-85ac-d1bd199c00fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668560 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4nq8\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-kube-api-access-b4nq8\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668583 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-envoy\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668629 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-credential-socket\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668696 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-token\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668900 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-certs\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.668999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.668916 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-socket\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.669185 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.669154 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-data\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.669293 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.669239 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-credential-socket\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.669426 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.669405 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54af582b-54c5-40ac-85ac-d1bd199c00fc-istiod-ca-cert\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.671018 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.670992 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-envoy\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.671142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.671052 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-podinfo\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.677162 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.677137 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-token\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.677372 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.677351 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4nq8\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-kube-api-access-b4nq8\") pod \"e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:23.719461 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:23.719419 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:24.004588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:24.004541 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx"] Sep 08 16:35:24.005981 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:35:24.005953 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54af582b_54c5_40ac_85ac_d1bd199c00fc.slice/crio-f92d3b3cd7b4102406c470b0b58e4f72c045ae2553353080c92b793f332cc6d7 WatchSource:0}: Error finding container f92d3b3cd7b4102406c470b0b58e4f72c045ae2553353080c92b793f332cc6d7: Status 404 returned error can't find the container with id f92d3b3cd7b4102406c470b0b58e4f72c045ae2553353080c92b793f332cc6d7 Sep 08 16:35:24.008582 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:24.008549 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:35:24.008656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:24.008621 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:35:24.008694 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:24.008656 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:35:24.851872 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:24.851835 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" event={"ID":"54af582b-54c5-40ac-85ac-d1bd199c00fc","Type":"ContainerStarted","Data":"556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f"} Sep 08 16:35:24.852500 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:24.851886 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" event={"ID":"54af582b-54c5-40ac-85ac-d1bd199c00fc","Type":"ContainerStarted","Data":"f92d3b3cd7b4102406c470b0b58e4f72c045ae2553353080c92b793f332cc6d7"} Sep 08 16:35:24.889642 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:24.889582 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" podStartSLOduration=1.889566017 podStartE2EDuration="1.889566017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:24.888386429 +0000 UTC m=+1358.483385206" watchObservedRunningTime="2026-09-08 16:35:24.889566017 +0000 UTC m=+1358.484564776" Sep 08 16:35:25.720597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:25.720555 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:25.725570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:25.725542 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:25.858301 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:25.858269 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:25.859210 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:25.859192 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:28.018940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.018908 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r"] Sep 08 16:35:28.022544 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.022521 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.027542 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.027514 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-8ded33-serving-cert\"" Sep 08 16:35:28.041003 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.040968 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r"] Sep 08 16:35:28.110777 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.110728 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7l5\" (UniqueName: \"kubernetes.io/projected/2deda381-a97c-4ba5-b940-49c20908d299-kube-api-access-nr7l5\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.110985 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.110819 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.110985 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.110937 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.110985 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.110983 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deda381-a97c-4ba5-b940-49c20908d299-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.212045 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.212007 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.212228 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.212081 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.212228 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:35:28.212170 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-serving-cert: secret "maas-api-e2e-shared-b-w2-8ded33-serving-cert" not found Sep 08 16:35:28.212343 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:35:28.212235 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls podName:2deda381-a97c-4ba5-b940-49c20908d299 nodeName:}" failed. No retries permitted until 2026-09-08 16:35:28.712214509 +0000 UTC m=+1362.307213269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls") pod "maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" (UID: "2deda381-a97c-4ba5-b940-49c20908d299") : secret "maas-api-e2e-shared-b-w2-8ded33-serving-cert" not found Sep 08 16:35:28.212343 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.212271 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deda381-a97c-4ba5-b940-49c20908d299-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.212429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.212372 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7l5\" (UniqueName: \"kubernetes.io/projected/2deda381-a97c-4ba5-b940-49c20908d299-kube-api-access-nr7l5\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.212927 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.212900 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deda381-a97c-4ba5-b940-49c20908d299-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.214604 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.214576 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.225228 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.225195 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7l5\" (UniqueName: \"kubernetes.io/projected/2deda381-a97c-4ba5-b940-49c20908d299-kube-api-access-nr7l5\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.719075 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.719035 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.721617 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.721586 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:28.933548 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:28.933510 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:29.232573 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:35:29.232542 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deda381_a97c_4ba5_b940_49c20908d299.slice/crio-cb962d3c774ca30aced262ca12fde1c2a91be758aeb4b3e9953e5e94949f98ca WatchSource:0}: Error finding container cb962d3c774ca30aced262ca12fde1c2a91be758aeb4b3e9953e5e94949f98ca: Status 404 returned error can't find the container with id cb962d3c774ca30aced262ca12fde1c2a91be758aeb4b3e9953e5e94949f98ca Sep 08 16:35:29.232962 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:29.232938 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r"] Sep 08 16:35:29.234434 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:29.234416 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:35:29.884179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:29.884140 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" event={"ID":"2deda381-a97c-4ba5-b940-49c20908d299","Type":"ContainerStarted","Data":"39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38"} Sep 08 16:35:29.884385 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:29.884196 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" event={"ID":"2deda381-a97c-4ba5-b940-49c20908d299","Type":"ContainerStarted","Data":"cb962d3c774ca30aced262ca12fde1c2a91be758aeb4b3e9953e5e94949f98ca"} Sep 08 16:35:29.884385 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:29.884329 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:29.919796 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:29.919684 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" podStartSLOduration=2.487796962 podStartE2EDuration="2.919668134s" totalImagesPullingTime="431.871172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:29.915457049 +0000 UTC m=+1363.510455828" watchObservedRunningTime="2026-09-08 16:35:29.919668134 +0000 UTC m=+1363.514666893" Sep 08 16:35:35.895932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:35.895896 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:35:44.979642 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:44.979544 2555 generic.go:356] "Generic (PLEG): container finished" podID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerID="556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f" exitCode=0 Sep 08 16:35:44.979642 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:44.979626 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" event={"ID":"54af582b-54c5-40ac-85ac-d1bd199c00fc","Type":"ContainerDied","Data":"556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f"} Sep 08 16:35:44.980165 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:44.980150 2555 scope.go:117] "RemoveContainer" containerID="556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f" Sep 08 16:35:44.981195 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:44.981167 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:35:44.981284 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:44.981232 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:35:44.981329 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:44.981281 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:35:45.987886 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:45.987855 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" event={"ID":"54af582b-54c5-40ac-85ac-d1bd199c00fc","Type":"ContainerStarted","Data":"482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4"} Sep 08 16:35:46.720590 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:46.720544 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:46.720820 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:46.720778 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 08 16:35:46.720900 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:46.720840 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 08 16:35:47.720536 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:47.720501 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 08 16:35:47.720937 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:47.720556 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 08 16:35:48.720915 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:48.720876 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 08 16:35:48.721464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:48.720957 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 08 16:35:49.720077 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:49.720040 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 08 16:35:49.720275 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:49.720102 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 08 16:35:50.719909 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:50.719870 2555 patch_prober.go:28] interesting pod/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 08 16:35:50.720352 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:50.719947 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 08 16:35:51.724330 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:51.724296 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:51.724761 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:51.724613 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:51.725323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:51.725306 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:35:52.381114 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.381080 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:52.384597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.384578 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:52.387438 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.387413 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wdbqb\"" Sep 08 16:35:52.392239 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.392204 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:52.455502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.455467 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4v6\" (UniqueName: \"kubernetes.io/projected/67c45313-5b94-45d5-b9c2-4921331526ae-kube-api-access-zl4v6\") pod \"test-curl-7899-139667675579712\" (UID: \"67c45313-5b94-45d5-b9c2-4921331526ae\") " pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:52.556915 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.556878 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4v6\" (UniqueName: \"kubernetes.io/projected/67c45313-5b94-45d5-b9c2-4921331526ae-kube-api-access-zl4v6\") pod \"test-curl-7899-139667675579712\" (UID: \"67c45313-5b94-45d5-b9c2-4921331526ae\") " pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:52.566639 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.566611 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4v6\" (UniqueName: \"kubernetes.io/projected/67c45313-5b94-45d5-b9c2-4921331526ae-kube-api-access-zl4v6\") pod \"test-curl-7899-139667675579712\" (UID: \"67c45313-5b94-45d5-b9c2-4921331526ae\") " pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:52.695903 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.695807 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:52.969544 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:52.969514 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:52.970916 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:35:52.970885 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67c45313_5b94_45d5_b9c2_4921331526ae.slice/crio-dfce4f17a5cea66f48288ad5a0d552be805006252d7457a09f6053ca5a22396c WatchSource:0}: Error finding container dfce4f17a5cea66f48288ad5a0d552be805006252d7457a09f6053ca5a22396c: Status 404 returned error can't find the container with id dfce4f17a5cea66f48288ad5a0d552be805006252d7457a09f6053ca5a22396c Sep 08 16:35:53.036603 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:53.036557 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675579712" event={"ID":"67c45313-5b94-45d5-b9c2-4921331526ae","Type":"ContainerStarted","Data":"dfce4f17a5cea66f48288ad5a0d552be805006252d7457a09f6053ca5a22396c"} Sep 08 16:35:54.044528 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:54.044490 2555 generic.go:356] "Generic (PLEG): container finished" podID="67c45313-5b94-45d5-b9c2-4921331526ae" containerID="cbc167cbc809aeb425e18fbd5e8100812f0ffc0426b7c3e046e20a4482332b45" exitCode=0 Sep 08 16:35:54.044919 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:54.044590 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675579712" event={"ID":"67c45313-5b94-45d5-b9c2-4921331526ae","Type":"ContainerDied","Data":"cbc167cbc809aeb425e18fbd5e8100812f0ffc0426b7c3e046e20a4482332b45"} Sep 08 16:35:55.180021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.179996 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:55.226716 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.226659 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7899-139667675579712_67c45313-5b94-45d5-b9c2-4921331526ae/test-curl-7899-139667675579712/0.log" Sep 08 16:35:55.250207 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.250170 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:55.256263 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.256216 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:55.285693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.285659 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl4v6\" (UniqueName: \"kubernetes.io/projected/67c45313-5b94-45d5-b9c2-4921331526ae-kube-api-access-zl4v6\") pod \"67c45313-5b94-45d5-b9c2-4921331526ae\" (UID: \"67c45313-5b94-45d5-b9c2-4921331526ae\") " Sep 08 16:35:55.287963 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.287937 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c45313-5b94-45d5-b9c2-4921331526ae-kube-api-access-zl4v6" (OuterVolumeSpecName: "kube-api-access-zl4v6") pod "67c45313-5b94-45d5-b9c2-4921331526ae" (UID: "67c45313-5b94-45d5-b9c2-4921331526ae"). InnerVolumeSpecName "kube-api-access-zl4v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:35:55.375225 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.375184 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:55.375637 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.375622 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67c45313-5b94-45d5-b9c2-4921331526ae" containerName="test-curl-7899-139667675579712" Sep 08 16:35:55.375682 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.375639 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67c45313-5b94-45d5-b9c2-4921331526ae" containerName="test-curl-7899-139667675579712" Sep 08 16:35:55.375728 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.375719 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67c45313-5b94-45d5-b9c2-4921331526ae" containerName="test-curl-7899-139667675579712" Sep 08 16:35:55.377841 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.377824 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:55.386597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.386570 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4v6\" (UniqueName: \"kubernetes.io/projected/67c45313-5b94-45d5-b9c2-4921331526ae-kube-api-access-zl4v6\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:55.388303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.388277 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:55.487633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.487575 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fsph\" (UniqueName: \"kubernetes.io/projected/7027ee31-575c-447a-9a83-f8e5ea7175b8-kube-api-access-9fsph\") pod \"test-curl-7899-139667675579712\" (UID: \"7027ee31-575c-447a-9a83-f8e5ea7175b8\") " pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:55.588206 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.588159 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fsph\" (UniqueName: \"kubernetes.io/projected/7027ee31-575c-447a-9a83-f8e5ea7175b8-kube-api-access-9fsph\") pod \"test-curl-7899-139667675579712\" (UID: \"7027ee31-575c-447a-9a83-f8e5ea7175b8\") " pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:55.599590 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.599560 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fsph\" (UniqueName: \"kubernetes.io/projected/7027ee31-575c-447a-9a83-f8e5ea7175b8-kube-api-access-9fsph\") pod \"test-curl-7899-139667675579712\" (UID: \"7027ee31-575c-447a-9a83-f8e5ea7175b8\") " pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:55.688528 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.688430 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:55.979220 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:55.979186 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:55.981179 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:35:55.981142 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7027ee31_575c_447a_9a83_f8e5ea7175b8.slice/crio-e3656e39e3bb357f9f7b5a0ce50ad387ce474dd3043fa24508a9de4e8aa54deb WatchSource:0}: Error finding container e3656e39e3bb357f9f7b5a0ce50ad387ce474dd3043fa24508a9de4e8aa54deb: Status 404 returned error can't find the container with id e3656e39e3bb357f9f7b5a0ce50ad387ce474dd3043fa24508a9de4e8aa54deb Sep 08 16:35:56.057155 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:56.057119 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675579712" event={"ID":"7027ee31-575c-447a-9a83-f8e5ea7175b8","Type":"ContainerStarted","Data":"e3656e39e3bb357f9f7b5a0ce50ad387ce474dd3043fa24508a9de4e8aa54deb"} Sep 08 16:35:56.058507 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:56.058487 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:56.058507 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:56.058500 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfce4f17a5cea66f48288ad5a0d552be805006252d7457a09f6053ca5a22396c" Sep 08 16:35:56.076229 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:56.076184 2555 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7899-139667675579712" oldPodUID="67c45313-5b94-45d5-b9c2-4921331526ae" podUID="7027ee31-575c-447a-9a83-f8e5ea7175b8" Sep 08 16:35:56.938595 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:56.938502 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67c45313-5b94-45d5-b9c2-4921331526ae" path="/var/lib/kubelet/pods/67c45313-5b94-45d5-b9c2-4921331526ae/volumes" Sep 08 16:35:57.066495 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:57.066457 2555 generic.go:356] "Generic (PLEG): container finished" podID="7027ee31-575c-447a-9a83-f8e5ea7175b8" containerID="c62f142e2c3de5b1ae28915cc42e9db013669202b3665e1996c21ba7971f6941" exitCode=0 Sep 08 16:35:57.066679 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:57.066547 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675579712" event={"ID":"7027ee31-575c-447a-9a83-f8e5ea7175b8","Type":"ContainerDied","Data":"c62f142e2c3de5b1ae28915cc42e9db013669202b3665e1996c21ba7971f6941"} Sep 08 16:35:58.205511 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.205486 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:35:58.233531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.233492 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7899-139667675579712_7027ee31-575c-447a-9a83-f8e5ea7175b8/test-curl-7899-139667675579712/0.log" Sep 08 16:35:58.268482 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.268448 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:58.278716 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.278684 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-139667675579712"] Sep 08 16:35:58.314453 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.314412 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fsph\" (UniqueName: \"kubernetes.io/projected/7027ee31-575c-447a-9a83-f8e5ea7175b8-kube-api-access-9fsph\") pod \"7027ee31-575c-447a-9a83-f8e5ea7175b8\" (UID: \"7027ee31-575c-447a-9a83-f8e5ea7175b8\") " Sep 08 16:35:58.316481 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.316452 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7027ee31-575c-447a-9a83-f8e5ea7175b8-kube-api-access-9fsph" (OuterVolumeSpecName: "kube-api-access-9fsph") pod "7027ee31-575c-447a-9a83-f8e5ea7175b8" (UID: "7027ee31-575c-447a-9a83-f8e5ea7175b8"). InnerVolumeSpecName "kube-api-access-9fsph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:35:58.400038 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.399988 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:35:58.400433 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.400419 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7027ee31-575c-447a-9a83-f8e5ea7175b8" containerName="test-curl-7899-139667675579712" Sep 08 16:35:58.400485 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.400434 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7027ee31-575c-447a-9a83-f8e5ea7175b8" containerName="test-curl-7899-139667675579712" Sep 08 16:35:58.400523 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.400518 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7027ee31-575c-447a-9a83-f8e5ea7175b8" containerName="test-curl-7899-139667675579712" Sep 08 16:35:58.402895 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.402877 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:35:58.414805 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.414769 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:35:58.415419 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.415392 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fsph\" (UniqueName: \"kubernetes.io/projected/7027ee31-575c-447a-9a83-f8e5ea7175b8-kube-api-access-9fsph\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:35:58.516677 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.516617 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsz5\" (UniqueName: \"kubernetes.io/projected/9752f324-1e70-42fa-95c5-5e3f95ac695e-kube-api-access-rdsz5\") pod \"test-curl-7899-139667675580288\" (UID: \"9752f324-1e70-42fa-95c5-5e3f95ac695e\") " pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:35:58.617306 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.617265 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsz5\" (UniqueName: \"kubernetes.io/projected/9752f324-1e70-42fa-95c5-5e3f95ac695e-kube-api-access-rdsz5\") pod \"test-curl-7899-139667675580288\" (UID: \"9752f324-1e70-42fa-95c5-5e3f95ac695e\") " pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:35:58.625974 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.625944 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsz5\" (UniqueName: \"kubernetes.io/projected/9752f324-1e70-42fa-95c5-5e3f95ac695e-kube-api-access-rdsz5\") pod \"test-curl-7899-139667675580288\" (UID: \"9752f324-1e70-42fa-95c5-5e3f95ac695e\") " pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:35:58.712381 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.712326 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:35:58.940231 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.940133 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7027ee31-575c-447a-9a83-f8e5ea7175b8" path="/var/lib/kubelet/pods/7027ee31-575c-447a-9a83-f8e5ea7175b8/volumes" Sep 08 16:35:58.999212 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:58.999185 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:35:59.001053 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:35:59.001028 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9752f324_1e70_42fa_95c5_5e3f95ac695e.slice/crio-7502b67c038b61846b5769476dffe01522ab4bb0632a94a9d4f949afa6bff101 WatchSource:0}: Error finding container 7502b67c038b61846b5769476dffe01522ab4bb0632a94a9d4f949afa6bff101: Status 404 returned error can't find the container with id 7502b67c038b61846b5769476dffe01522ab4bb0632a94a9d4f949afa6bff101 Sep 08 16:35:59.079150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:59.079113 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675580288" event={"ID":"9752f324-1e70-42fa-95c5-5e3f95ac695e","Type":"ContainerStarted","Data":"7502b67c038b61846b5769476dffe01522ab4bb0632a94a9d4f949afa6bff101"} Sep 08 16:35:59.080463 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:59.080446 2555 scope.go:117] "RemoveContainer" containerID="c62f142e2c3de5b1ae28915cc42e9db013669202b3665e1996c21ba7971f6941" Sep 08 16:35:59.080559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:35:59.080464 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675579712" Sep 08 16:36:01.095710 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:01.095666 2555 generic.go:356] "Generic (PLEG): container finished" podID="9752f324-1e70-42fa-95c5-5e3f95ac695e" containerID="54684a02869d19b1f28dba8e4762c1ad96656c7850bf0f66540303fca148e9a7" exitCode=0 Sep 08 16:36:01.096138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:01.095757 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675580288" event={"ID":"9752f324-1e70-42fa-95c5-5e3f95ac695e","Type":"ContainerDied","Data":"54684a02869d19b1f28dba8e4762c1ad96656c7850bf0f66540303fca148e9a7"} Sep 08 16:36:02.235595 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.235568 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:02.256439 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.256394 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7899-139667675580288_9752f324-1e70-42fa-95c5-5e3f95ac695e/test-curl-7899-139667675580288/0.log" Sep 08 16:36:02.278545 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.278500 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:36:02.282838 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.282805 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:36:02.352141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.352057 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdsz5\" (UniqueName: \"kubernetes.io/projected/9752f324-1e70-42fa-95c5-5e3f95ac695e-kube-api-access-rdsz5\") pod \"9752f324-1e70-42fa-95c5-5e3f95ac695e\" (UID: \"9752f324-1e70-42fa-95c5-5e3f95ac695e\") " Sep 08 16:36:02.354181 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.354156 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9752f324-1e70-42fa-95c5-5e3f95ac695e-kube-api-access-rdsz5" (OuterVolumeSpecName: "kube-api-access-rdsz5") pod "9752f324-1e70-42fa-95c5-5e3f95ac695e" (UID: "9752f324-1e70-42fa-95c5-5e3f95ac695e"). InnerVolumeSpecName "kube-api-access-rdsz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:36:02.403509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.403464 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:36:02.403932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.403916 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9752f324-1e70-42fa-95c5-5e3f95ac695e" containerName="test-curl-7899-139667675580288" Sep 08 16:36:02.403976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.403934 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9752f324-1e70-42fa-95c5-5e3f95ac695e" containerName="test-curl-7899-139667675580288" Sep 08 16:36:02.404019 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.404009 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9752f324-1e70-42fa-95c5-5e3f95ac695e" containerName="test-curl-7899-139667675580288" Sep 08 16:36:02.406937 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.406920 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:02.412100 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.412073 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:36:02.453719 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.453675 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdsz5\" (UniqueName: \"kubernetes.io/projected/9752f324-1e70-42fa-95c5-5e3f95ac695e-kube-api-access-rdsz5\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:36:02.554516 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.554473 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmdl\" (UniqueName: \"kubernetes.io/projected/b8c7254f-a92e-4648-912c-5b548fb7cea4-kube-api-access-4jmdl\") pod \"test-curl-7899-139667675580288\" (UID: \"b8c7254f-a92e-4648-912c-5b548fb7cea4\") " pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:02.655182 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.655082 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmdl\" (UniqueName: \"kubernetes.io/projected/b8c7254f-a92e-4648-912c-5b548fb7cea4-kube-api-access-4jmdl\") pod \"test-curl-7899-139667675580288\" (UID: \"b8c7254f-a92e-4648-912c-5b548fb7cea4\") " pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:02.663709 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.663684 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmdl\" (UniqueName: \"kubernetes.io/projected/b8c7254f-a92e-4648-912c-5b548fb7cea4-kube-api-access-4jmdl\") pod \"test-curl-7899-139667675580288\" (UID: \"b8c7254f-a92e-4648-912c-5b548fb7cea4\") " pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:02.717599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.717560 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:02.937178 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.937074 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9752f324-1e70-42fa-95c5-5e3f95ac695e" path="/var/lib/kubelet/pods/9752f324-1e70-42fa-95c5-5e3f95ac695e/volumes" Sep 08 16:36:02.989360 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:02.989326 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:36:02.990699 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:36:02.990674 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8c7254f_a92e_4648_912c_5b548fb7cea4.slice/crio-11f48bfced79ce0d5c39f08d66878ddc774d88ef882202c1ef3b377fe02ac8c4 WatchSource:0}: Error finding container 11f48bfced79ce0d5c39f08d66878ddc774d88ef882202c1ef3b377fe02ac8c4: Status 404 returned error can't find the container with id 11f48bfced79ce0d5c39f08d66878ddc774d88ef882202c1ef3b377fe02ac8c4 Sep 08 16:36:03.108399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:03.108371 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:03.108609 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:03.108422 2555 scope.go:117] "RemoveContainer" containerID="54684a02869d19b1f28dba8e4762c1ad96656c7850bf0f66540303fca148e9a7" Sep 08 16:36:03.109828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:03.109804 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675580288" event={"ID":"b8c7254f-a92e-4648-912c-5b548fb7cea4","Type":"ContainerStarted","Data":"11f48bfced79ce0d5c39f08d66878ddc774d88ef882202c1ef3b377fe02ac8c4"} Sep 08 16:36:04.120494 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:04.120454 2555 generic.go:356] "Generic (PLEG): container finished" podID="b8c7254f-a92e-4648-912c-5b548fb7cea4" containerID="30e897674b79b10b46bde52b0fc9dbab052e885a944949c0e15b525b6fba4caa" exitCode=0 Sep 08 16:36:04.120954 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:04.120540 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7899-139667675580288" event={"ID":"b8c7254f-a92e-4648-912c-5b548fb7cea4","Type":"ContainerDied","Data":"30e897674b79b10b46bde52b0fc9dbab052e885a944949c0e15b525b6fba4caa"} Sep 08 16:36:05.262802 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:05.262772 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:05.287701 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:05.287667 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7899-139667675580288_b8c7254f-a92e-4648-912c-5b548fb7cea4/test-curl-7899-139667675580288/0.log" Sep 08 16:36:05.311062 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:05.311021 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:36:05.317316 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:05.317282 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7899-139667675580288"] Sep 08 16:36:05.383544 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:05.383435 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jmdl\" (UniqueName: \"kubernetes.io/projected/b8c7254f-a92e-4648-912c-5b548fb7cea4-kube-api-access-4jmdl\") pod \"b8c7254f-a92e-4648-912c-5b548fb7cea4\" (UID: \"b8c7254f-a92e-4648-912c-5b548fb7cea4\") " Sep 08 16:36:05.385649 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:05.385611 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c7254f-a92e-4648-912c-5b548fb7cea4-kube-api-access-4jmdl" (OuterVolumeSpecName: "kube-api-access-4jmdl") pod "b8c7254f-a92e-4648-912c-5b548fb7cea4" (UID: "b8c7254f-a92e-4648-912c-5b548fb7cea4"). InnerVolumeSpecName "kube-api-access-4jmdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:36:05.485196 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:05.485153 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jmdl\" (UniqueName: \"kubernetes.io/projected/b8c7254f-a92e-4648-912c-5b548fb7cea4-kube-api-access-4jmdl\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:36:06.135731 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.135701 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11f48bfced79ce0d5c39f08d66878ddc774d88ef882202c1ef3b377fe02ac8c4" Sep 08 16:36:06.135731 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.135709 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:06.150187 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:36:06.150154 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-139667675580288\" is forbidden: User \"system:node:ip-10-0-139-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.ec2.internal' and this object" podUID="b8c7254f-a92e-4648-912c-5b548fb7cea4" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:06.254695 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.254659 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq"] Sep 08 16:36:06.255118 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.255083 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c7254f-a92e-4648-912c-5b548fb7cea4" containerName="test-curl-7899-139667675580288" Sep 08 16:36:06.255174 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.255119 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c7254f-a92e-4648-912c-5b548fb7cea4" containerName="test-curl-7899-139667675580288" Sep 08 16:36:06.255225 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.255214 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c7254f-a92e-4648-912c-5b548fb7cea4" containerName="test-curl-7899-139667675580288" Sep 08 16:36:06.258305 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.258285 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.260776 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:36:06.260743 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-139667675580288\" is forbidden: User \"system:node:ip-10-0-139-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.ec2.internal' and this object" podUID="b8c7254f-a92e-4648-912c-5b548fb7cea4" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:06.261078 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.261049 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1fa1601c-openshift-default-dockercfg-rlvbd\"" Sep 08 16:36:06.275490 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.275455 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq"] Sep 08 16:36:06.395286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395175 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istiod-ca-cert\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395230 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-envoy\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395468 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395322 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-credential-socket\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395468 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395391 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-socket\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395468 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395418 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-token\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395468 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395451 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-certs\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395494 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-data\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395561 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6rp\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-kube-api-access-xc6rp\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.395611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.395605 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-podinfo\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496156 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496110 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-envoy\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496156 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496160 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-credential-socket\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496393 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496210 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-socket\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496393 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496270 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-token\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496393 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496321 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-certs\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496393 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496345 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-data\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496579 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496393 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6rp\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-kube-api-access-xc6rp\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496579 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496451 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-podinfo\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496579 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496496 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istiod-ca-cert\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496694 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-socket\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496718 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-credential-socket\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.496889 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.496838 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-certs\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.497058 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.497037 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-data\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.497343 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.497319 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istiod-ca-cert\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.498724 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.498696 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-envoy\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.499076 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.499058 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-podinfo\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.505106 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.505082 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-token\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.505232 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.505213 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6rp\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-kube-api-access-xc6rp\") pod \"e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.569334 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.569291 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:06.860051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.860021 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq"] Sep 08 16:36:06.861733 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:36:06.861697 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc20a25_8dd6_433f_ac8d_5d6934ea6b2f.slice/crio-79ad15d72737dc245ff9275df7128462861bda88a60650bccd7ee7e1405b58c5 WatchSource:0}: Error finding container 79ad15d72737dc245ff9275df7128462861bda88a60650bccd7ee7e1405b58c5: Status 404 returned error can't find the container with id 79ad15d72737dc245ff9275df7128462861bda88a60650bccd7ee7e1405b58c5 Sep 08 16:36:06.864331 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.864234 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:36:06.864412 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.864389 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:36:06.864456 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.864439 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:36:06.937191 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:06.937159 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c7254f-a92e-4648-912c-5b548fb7cea4" path="/var/lib/kubelet/pods/b8c7254f-a92e-4648-912c-5b548fb7cea4/volumes" Sep 08 16:36:06.937597 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:36:06.937565 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7899-139667675580288\" is forbidden: User \"system:node:ip-10-0-139-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.ec2.internal' and this object" podUID="b8c7254f-a92e-4648-912c-5b548fb7cea4" pod="opendatahub/test-curl-7899-139667675580288" Sep 08 16:36:07.144014 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:07.143918 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" event={"ID":"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f","Type":"ContainerStarted","Data":"eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0"} Sep 08 16:36:07.144014 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:07.143976 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" event={"ID":"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f","Type":"ContainerStarted","Data":"79ad15d72737dc245ff9275df7128462861bda88a60650bccd7ee7e1405b58c5"} Sep 08 16:36:07.166599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:07.166535 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podStartSLOduration=1.166516651 podStartE2EDuration="1.166516651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:07.164394908 +0000 UTC m=+1400.759393684" watchObservedRunningTime="2026-09-08 16:36:07.166516651 +0000 UTC m=+1400.761515410" Sep 08 16:36:07.570204 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:07.570102 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:07.575569 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:07.575542 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:08.150169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:08.150141 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:08.151226 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:08.151206 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:30.303176 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:30.303140 2555 generic.go:356] "Generic (PLEG): container finished" podID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerID="eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0" exitCode=0 Sep 08 16:36:30.303763 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:30.303217 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" event={"ID":"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f","Type":"ContainerDied","Data":"eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0"} Sep 08 16:36:30.303851 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:30.303829 2555 scope.go:117] "RemoveContainer" containerID="eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0" Sep 08 16:36:30.305017 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:30.304984 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:36:30.305107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:30.305063 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:36:30.305107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:30.305094 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:36:31.311141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:31.311105 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" event={"ID":"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f","Type":"ContainerStarted","Data":"05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c"} Sep 08 16:36:31.570476 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:31.570377 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:31.570658 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:31.570632 2555 patch_prober.go:28] interesting pod/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 08 16:36:31.570721 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:31.570695 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 08 16:36:32.570170 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:32.570127 2555 patch_prober.go:28] interesting pod/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 08 16:36:32.570697 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:32.570195 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 08 16:36:33.569737 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:33.569694 2555 patch_prober.go:28] interesting pod/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 08 16:36:33.569947 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:33.569767 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 08 16:36:34.570421 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:34.570379 2555 patch_prober.go:28] interesting pod/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 08 16:36:34.570841 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:34.570452 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 08 16:36:35.569897 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:35.569856 2555 patch_prober.go:28] interesting pod/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 08 16:36:35.570089 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:35.569916 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 08 16:36:36.569844 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:36.569802 2555 patch_prober.go:28] interesting pod/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 08 16:36:36.570299 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:36.569872 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 08 16:36:37.574661 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:37.574624 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:37.575135 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:37.574939 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:36:37.575647 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:36:37.575628 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:37:40.342073 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.342028 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7902-f55b94"] Sep 08 16:37:40.345151 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.345124 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:40.348331 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.348312 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wdbqb\"" Sep 08 16:37:40.351968 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.351944 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7902-f55b94"] Sep 08 16:37:40.433374 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.433335 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27tb\" (UniqueName: \"kubernetes.io/projected/b11c9673-f909-47b6-872c-5071f0ca103c-kube-api-access-k27tb\") pod \"mt-curl-7902-f55b94\" (UID: \"b11c9673-f909-47b6-872c-5071f0ca103c\") " pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:40.534627 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.534587 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27tb\" (UniqueName: \"kubernetes.io/projected/b11c9673-f909-47b6-872c-5071f0ca103c-kube-api-access-k27tb\") pod \"mt-curl-7902-f55b94\" (UID: \"b11c9673-f909-47b6-872c-5071f0ca103c\") " pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:40.543617 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.543591 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27tb\" (UniqueName: \"kubernetes.io/projected/b11c9673-f909-47b6-872c-5071f0ca103c-kube-api-access-k27tb\") pod \"mt-curl-7902-f55b94\" (UID: \"b11c9673-f909-47b6-872c-5071f0ca103c\") " pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:40.656376 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.656327 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:40.942943 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:40.942917 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7902-f55b94"] Sep 08 16:37:40.946532 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:37:40.946503 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb11c9673_f909_47b6_872c_5071f0ca103c.slice/crio-5056f7d318acfff7e91e4bc427a2620b42a108da2a34326de920ead6456fd986 WatchSource:0}: Error finding container 5056f7d318acfff7e91e4bc427a2620b42a108da2a34326de920ead6456fd986: Status 404 returned error can't find the container with id 5056f7d318acfff7e91e4bc427a2620b42a108da2a34326de920ead6456fd986 Sep 08 16:37:41.737494 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:41.737452 2555 generic.go:356] "Generic (PLEG): container finished" podID="b11c9673-f909-47b6-872c-5071f0ca103c" containerID="35a12a9b0e8794b847f40ecd3e8076ef9949ffef2c8482e371f5c05dea5f83ac" exitCode=0 Sep 08 16:37:41.737956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:41.737581 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7902-f55b94" event={"ID":"b11c9673-f909-47b6-872c-5071f0ca103c","Type":"ContainerDied","Data":"35a12a9b0e8794b847f40ecd3e8076ef9949ffef2c8482e371f5c05dea5f83ac"} Sep 08 16:37:41.737956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:41.737639 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7902-f55b94" event={"ID":"b11c9673-f909-47b6-872c-5071f0ca103c","Type":"ContainerStarted","Data":"5056f7d318acfff7e91e4bc427a2620b42a108da2a34326de920ead6456fd986"} Sep 08 16:37:42.882409 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:42.882380 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:42.916924 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:42.916890 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7902-f55b94_b11c9673-f909-47b6-872c-5071f0ca103c/mt-curl-7902-f55b94/0.log" Sep 08 16:37:42.939415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:42.939369 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7902-f55b94"] Sep 08 16:37:42.942328 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:42.942299 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7902-f55b94"] Sep 08 16:37:42.958480 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:42.958447 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k27tb\" (UniqueName: \"kubernetes.io/projected/b11c9673-f909-47b6-872c-5071f0ca103c-kube-api-access-k27tb\") pod \"b11c9673-f909-47b6-872c-5071f0ca103c\" (UID: \"b11c9673-f909-47b6-872c-5071f0ca103c\") " Sep 08 16:37:42.960633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:42.960604 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11c9673-f909-47b6-872c-5071f0ca103c-kube-api-access-k27tb" (OuterVolumeSpecName: "kube-api-access-k27tb") pod "b11c9673-f909-47b6-872c-5071f0ca103c" (UID: "b11c9673-f909-47b6-872c-5071f0ca103c"). InnerVolumeSpecName "kube-api-access-k27tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:37:43.059361 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:43.059235 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27tb\" (UniqueName: \"kubernetes.io/projected/b11c9673-f909-47b6-872c-5071f0ca103c-kube-api-access-k27tb\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:37:43.750888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:43.750855 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:43.751098 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:43.750895 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5056f7d318acfff7e91e4bc427a2620b42a108da2a34326de920ead6456fd986" Sep 08 16:37:43.761063 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:37:43.761032 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7902-f55b94\" is forbidden: User \"system:node:ip-10-0-139-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.ec2.internal' and this object" podUID="b11c9673-f909-47b6-872c-5071f0ca103c" pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:44.300280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:44.300232 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl"] Sep 08 16:37:44.319811 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:37:44.319768 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7902-f55b94\" is forbidden: User \"system:node:ip-10-0-139-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.ec2.internal' and this object" podUID="b11c9673-f909-47b6-872c-5071f0ca103c" pod="opendatahub/mt-curl-7902-f55b94" Sep 08 16:37:44.937714 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:44.937615 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11c9673-f909-47b6-872c-5071f0ca103c" path="/var/lib/kubelet/pods/b11c9673-f909-47b6-872c-5071f0ca103c/volumes" Sep 08 16:37:59.320413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.320363 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" podUID="77f45bfb-a143-4ccf-a7e9-f618ef161560" containerName="main" containerID="cri-o://caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642" gracePeriod=45 Sep 08 16:37:59.578835 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.578744 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:37:59.728854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.728814 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-model-cache\") pod \"77f45bfb-a143-4ccf-a7e9-f618ef161560\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " Sep 08 16:37:59.728854 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.728854 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t2f7\" (UniqueName: \"kubernetes.io/projected/77f45bfb-a143-4ccf-a7e9-f618ef161560-kube-api-access-4t2f7\") pod \"77f45bfb-a143-4ccf-a7e9-f618ef161560\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " Sep 08 16:37:59.729065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.728903 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-tmp-dir\") pod \"77f45bfb-a143-4ccf-a7e9-f618ef161560\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " Sep 08 16:37:59.729065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.728948 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f45bfb-a143-4ccf-a7e9-f618ef161560-tls-certs\") pod \"77f45bfb-a143-4ccf-a7e9-f618ef161560\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " Sep 08 16:37:59.729065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729026 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-dshm\") pod \"77f45bfb-a143-4ccf-a7e9-f618ef161560\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " Sep 08 16:37:59.729065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729063 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-home\") pod \"77f45bfb-a143-4ccf-a7e9-f618ef161560\" (UID: \"77f45bfb-a143-4ccf-a7e9-f618ef161560\") " Sep 08 16:37:59.729273 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729086 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-model-cache" (OuterVolumeSpecName: "model-cache") pod "77f45bfb-a143-4ccf-a7e9-f618ef161560" (UID: "77f45bfb-a143-4ccf-a7e9-f618ef161560"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:37:59.729273 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729191 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77f45bfb-a143-4ccf-a7e9-f618ef161560" (UID: "77f45bfb-a143-4ccf-a7e9-f618ef161560"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:37:59.729402 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729377 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-home" (OuterVolumeSpecName: "home") pod "77f45bfb-a143-4ccf-a7e9-f618ef161560" (UID: "77f45bfb-a143-4ccf-a7e9-f618ef161560"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:37:59.729495 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729471 2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-home\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:37:59.729495 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729492 2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-model-cache\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:37:59.729698 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.729526 2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-tmp-dir\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:37:59.731202 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.731180 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-dshm" (OuterVolumeSpecName: "dshm") pod "77f45bfb-a143-4ccf-a7e9-f618ef161560" (UID: "77f45bfb-a143-4ccf-a7e9-f618ef161560"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:37:59.731333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.731313 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f45bfb-a143-4ccf-a7e9-f618ef161560-kube-api-access-4t2f7" (OuterVolumeSpecName: "kube-api-access-4t2f7") pod "77f45bfb-a143-4ccf-a7e9-f618ef161560" (UID: "77f45bfb-a143-4ccf-a7e9-f618ef161560"). InnerVolumeSpecName "kube-api-access-4t2f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:37:59.731380 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.731352 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f45bfb-a143-4ccf-a7e9-f618ef161560-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77f45bfb-a143-4ccf-a7e9-f618ef161560" (UID: "77f45bfb-a143-4ccf-a7e9-f618ef161560"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:37:59.831147 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.831045 2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77f45bfb-a143-4ccf-a7e9-f618ef161560-dshm\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:37:59.831147 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.831084 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2f7\" (UniqueName: \"kubernetes.io/projected/77f45bfb-a143-4ccf-a7e9-f618ef161560-kube-api-access-4t2f7\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:37:59.831147 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.831099 2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77f45bfb-a143-4ccf-a7e9-f618ef161560-tls-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:37:59.853826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.853781 2555 generic.go:356] "Generic (PLEG): container finished" podID="77f45bfb-a143-4ccf-a7e9-f618ef161560" containerID="caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642" exitCode=0 Sep 08 16:37:59.854037 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.853901 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" Sep 08 16:37:59.854098 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.854037 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" event={"ID":"77f45bfb-a143-4ccf-a7e9-f618ef161560","Type":"ContainerDied","Data":"caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642"} Sep 08 16:37:59.854098 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.854079 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl" event={"ID":"77f45bfb-a143-4ccf-a7e9-f618ef161560","Type":"ContainerDied","Data":"69c5bebba846372bfa26333ece151dadb696760f57f41b422f96e3da4cfb95d5"} Sep 08 16:37:59.854210 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.854101 2555 scope.go:117] "RemoveContainer" containerID="caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642" Sep 08 16:37:59.867891 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.867867 2555 scope.go:117] "RemoveContainer" containerID="caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642" Sep 08 16:37:59.868387 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:37:59.868342 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642\": container with ID starting with caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642 not found: ID does not exist" containerID="caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642" Sep 08 16:37:59.868525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.868395 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642"} err="failed to get container status \"caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642\": rpc error: code = NotFound desc = could not find container \"caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642\": container with ID starting with caca969ee416600592c5cfdb62cd66fb06e9c222f05f3b20e408be539dc69642 not found: ID does not exist" Sep 08 16:37:59.885463 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.885425 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl"] Sep 08 16:37:59.888626 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:37:59.888595 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bce20b/auth-test-model-bce20b-kserve-78b7cc9cf5-qrjkl"] Sep 08 16:38:00.944570 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:38:00.944522 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f45bfb-a143-4ccf-a7e9-f618ef161560" path="/var/lib/kubelet/pods/77f45bfb-a143-4ccf-a7e9-f618ef161560/volumes" Sep 08 16:39:05.893824 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:05.893787 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq"] Sep 08 16:39:05.894437 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:05.894039 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" containerID="cri-o://05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c" gracePeriod=30 Sep 08 16:39:07.577574 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:07.577527 2555 patch_prober.go:28] interesting pod/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 16:39:07.577986 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:07.577606 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 16:39:11.151169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.151140 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:39:11.224961 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.224876 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc6rp\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-kube-api-access-xc6rp\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.224961 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.224924 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-data\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.224961 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.224943 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-token\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.225264 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225072 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-certs\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.225264 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225183 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-credential-socket\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.225264 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225232 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istiod-ca-cert\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.225433 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225239 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:39:11.225433 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225293 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-envoy\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.225433 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225320 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-podinfo\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.225433 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225386 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-socket\") pod \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\" (UID: \"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f\") " Sep 08 16:39:11.225433 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225389 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:39:11.225723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225445 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:39:11.225723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225582 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:39:11.225723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225708 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:39:11.225893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225873 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.225893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225891 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.226002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225903 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.226002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225914 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.226002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.225924 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.227574 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.227514 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:39:11.227688 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.227605 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:39:11.227826 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.227803 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-kube-api-access-xc6rp" (OuterVolumeSpecName: "kube-api-access-xc6rp") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "kube-api-access-xc6rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:39:11.227978 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.227959 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" (UID: "7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:39:11.289327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.289293 2555 generic.go:356] "Generic (PLEG): container finished" podID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerID="05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c" exitCode=0 Sep 08 16:39:11.289502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.289343 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" Sep 08 16:39:11.289502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.289377 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" event={"ID":"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f","Type":"ContainerDied","Data":"05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c"} Sep 08 16:39:11.289502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.289455 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq" event={"ID":"7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f","Type":"ContainerDied","Data":"79ad15d72737dc245ff9275df7128462861bda88a60650bccd7ee7e1405b58c5"} Sep 08 16:39:11.289502 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.289478 2555 scope.go:117] "RemoveContainer" containerID="05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c" Sep 08 16:39:11.302952 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.302930 2555 scope.go:117] "RemoveContainer" containerID="eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0" Sep 08 16:39:11.312665 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.312642 2555 scope.go:117] "RemoveContainer" containerID="05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c" Sep 08 16:39:11.312966 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:39:11.312946 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c\": container with ID starting with 05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c not found: ID does not exist" containerID="05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c" Sep 08 16:39:11.313035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.312975 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c"} err="failed to get container status \"05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c\": rpc error: code = NotFound desc = could not find container \"05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c\": container with ID starting with 05520184253cdadb4c8863b2ba6d0eb317eddd236b8d47f3e0abb1deb6ba797c not found: ID does not exist" Sep 08 16:39:11.313035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.312992 2555 scope.go:117] "RemoveContainer" containerID="eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0" Sep 08 16:39:11.313201 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:39:11.313188 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0\": container with ID starting with eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0 not found: ID does not exist" containerID="eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0" Sep 08 16:39:11.313239 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.313205 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0"} err="failed to get container status \"eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0\": rpc error: code = NotFound desc = could not find container \"eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0\": container with ID starting with eea63e10f7cad236bef3aef19d3b4f0358b20a504b2bd6325deda6728a8438c0 not found: ID does not exist" Sep 08 16:39:11.320271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.320217 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq"] Sep 08 16:39:11.324152 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.324124 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1fa1601c-openshift-default-7c7b68f8dd-w5hlq"] Sep 08 16:39:11.327493 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.327463 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.327493 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.327488 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.327638 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.327497 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6rp\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-kube-api-access-xc6rp\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:11.327638 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:11.327509 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:12.937047 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:12.937009 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" path="/var/lib/kubelet/pods/7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f/volumes" Sep 08 16:39:29.969322 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969281 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7902-1abf81"] Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969682 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f45bfb-a143-4ccf-a7e9-f618ef161560" containerName="main" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969695 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f45bfb-a143-4ccf-a7e9-f618ef161560" containerName="main" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969716 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969721 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969728 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969734 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969751 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11c9673-f909-47b6-872c-5071f0ca103c" containerName="mt-curl-7902-f55b94" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969756 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11c9673-f909-47b6-872c-5071f0ca103c" containerName="mt-curl-7902-f55b94" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969820 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11c9673-f909-47b6-872c-5071f0ca103c" containerName="mt-curl-7902-f55b94" Sep 08 16:39:29.969828 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969828 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" Sep 08 16:39:29.970138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969836 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc20a25-8dd6-433f-ac8d-5d6934ea6b2f" containerName="istio-proxy" Sep 08 16:39:29.970138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.969843 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f45bfb-a143-4ccf-a7e9-f618ef161560" containerName="main" Sep 08 16:39:29.973014 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.972995 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-1abf81" Sep 08 16:39:29.975370 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.975347 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wdbqb\"" Sep 08 16:39:29.979592 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:29.979565 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7902-1abf81"] Sep 08 16:39:30.109277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:30.109201 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwks\" (UniqueName: \"kubernetes.io/projected/03b68dab-d358-454b-b620-1f14fe61b8d6-kube-api-access-htwks\") pod \"mt-curl-7902-1abf81\" (UID: \"03b68dab-d358-454b-b620-1f14fe61b8d6\") " pod="opendatahub/mt-curl-7902-1abf81" Sep 08 16:39:30.210272 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:30.210204 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwks\" (UniqueName: \"kubernetes.io/projected/03b68dab-d358-454b-b620-1f14fe61b8d6-kube-api-access-htwks\") pod \"mt-curl-7902-1abf81\" (UID: \"03b68dab-d358-454b-b620-1f14fe61b8d6\") " pod="opendatahub/mt-curl-7902-1abf81" Sep 08 16:39:30.218669 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:30.218636 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwks\" (UniqueName: \"kubernetes.io/projected/03b68dab-d358-454b-b620-1f14fe61b8d6-kube-api-access-htwks\") pod \"mt-curl-7902-1abf81\" (UID: \"03b68dab-d358-454b-b620-1f14fe61b8d6\") " pod="opendatahub/mt-curl-7902-1abf81" Sep 08 16:39:30.284379 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:30.284281 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-1abf81" Sep 08 16:39:30.557849 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:30.557822 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7902-1abf81"] Sep 08 16:39:30.560068 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:39:30.560038 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03b68dab_d358_454b_b620_1f14fe61b8d6.slice/crio-177782cdf4cb9d868ba212aed12a8cf9482f46b906ef170021eb4bf48e125c47 WatchSource:0}: Error finding container 177782cdf4cb9d868ba212aed12a8cf9482f46b906ef170021eb4bf48e125c47: Status 404 returned error can't find the container with id 177782cdf4cb9d868ba212aed12a8cf9482f46b906ef170021eb4bf48e125c47 Sep 08 16:39:31.413086 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:31.413045 2555 generic.go:356] "Generic (PLEG): container finished" podID="03b68dab-d358-454b-b620-1f14fe61b8d6" containerID="14fb9c41bb529f5750c62faa654c3942b465b5daac5014b0d5d33677ebd1822c" exitCode=0 Sep 08 16:39:31.413514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:31.413132 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7902-1abf81" event={"ID":"03b68dab-d358-454b-b620-1f14fe61b8d6","Type":"ContainerDied","Data":"14fb9c41bb529f5750c62faa654c3942b465b5daac5014b0d5d33677ebd1822c"} Sep 08 16:39:31.413514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:31.413178 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7902-1abf81" event={"ID":"03b68dab-d358-454b-b620-1f14fe61b8d6","Type":"ContainerStarted","Data":"177782cdf4cb9d868ba212aed12a8cf9482f46b906ef170021eb4bf48e125c47"} Sep 08 16:39:32.552889 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.552860 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-1abf81" Sep 08 16:39:32.599400 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.599372 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7902-1abf81_03b68dab-d358-454b-b620-1f14fe61b8d6/mt-curl-7902-1abf81/0.log" Sep 08 16:39:32.623929 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.623889 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7902-1abf81"] Sep 08 16:39:32.632631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.632597 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htwks\" (UniqueName: \"kubernetes.io/projected/03b68dab-d358-454b-b620-1f14fe61b8d6-kube-api-access-htwks\") pod \"03b68dab-d358-454b-b620-1f14fe61b8d6\" (UID: \"03b68dab-d358-454b-b620-1f14fe61b8d6\") " Sep 08 16:39:32.634928 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.634892 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b68dab-d358-454b-b620-1f14fe61b8d6-kube-api-access-htwks" (OuterVolumeSpecName: "kube-api-access-htwks") pod "03b68dab-d358-454b-b620-1f14fe61b8d6" (UID: "03b68dab-d358-454b-b620-1f14fe61b8d6"). InnerVolumeSpecName "kube-api-access-htwks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:39:32.636622 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.636601 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7902-1abf81"] Sep 08 16:39:32.733938 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.733842 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwks\" (UniqueName: \"kubernetes.io/projected/03b68dab-d358-454b-b620-1f14fe61b8d6-kube-api-access-htwks\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:32.749438 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.749400 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7902-0531d4"] Sep 08 16:39:32.749860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.749844 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b68dab-d358-454b-b620-1f14fe61b8d6" containerName="mt-curl-7902-1abf81" Sep 08 16:39:32.749904 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.749863 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b68dab-d358-454b-b620-1f14fe61b8d6" containerName="mt-curl-7902-1abf81" Sep 08 16:39:32.749946 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.749940 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b68dab-d358-454b-b620-1f14fe61b8d6" containerName="mt-curl-7902-1abf81" Sep 08 16:39:32.753089 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.753064 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:32.762008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.761981 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7902-0531d4"] Sep 08 16:39:32.835080 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.835039 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztc8\" (UniqueName: \"kubernetes.io/projected/6df0364c-8a9a-419f-bd48-a56993d52a2a-kube-api-access-2ztc8\") pod \"mt-curl-7902-0531d4\" (UID: \"6df0364c-8a9a-419f-bd48-a56993d52a2a\") " pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:32.935673 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.935638 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztc8\" (UniqueName: \"kubernetes.io/projected/6df0364c-8a9a-419f-bd48-a56993d52a2a-kube-api-access-2ztc8\") pod \"mt-curl-7902-0531d4\" (UID: \"6df0364c-8a9a-419f-bd48-a56993d52a2a\") " pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:32.937679 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.937648 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b68dab-d358-454b-b620-1f14fe61b8d6" path="/var/lib/kubelet/pods/03b68dab-d358-454b-b620-1f14fe61b8d6/volumes" Sep 08 16:39:32.945348 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:32.945319 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztc8\" (UniqueName: \"kubernetes.io/projected/6df0364c-8a9a-419f-bd48-a56993d52a2a-kube-api-access-2ztc8\") pod \"mt-curl-7902-0531d4\" (UID: \"6df0364c-8a9a-419f-bd48-a56993d52a2a\") " pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:33.063437 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:33.063343 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:33.349752 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:33.349726 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7902-0531d4"] Sep 08 16:39:33.425683 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:33.425657 2555 scope.go:117] "RemoveContainer" containerID="14fb9c41bb529f5750c62faa654c3942b465b5daac5014b0d5d33677ebd1822c" Sep 08 16:39:33.425683 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:33.425669 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-1abf81" Sep 08 16:39:33.427193 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:33.427161 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7902-0531d4" event={"ID":"6df0364c-8a9a-419f-bd48-a56993d52a2a","Type":"ContainerStarted","Data":"e7166e0aff9b65762537eb055635428a3a126b20d1c1c3c7ab55068a2b2d2157"} Sep 08 16:39:34.435385 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:34.435343 2555 generic.go:356] "Generic (PLEG): container finished" podID="6df0364c-8a9a-419f-bd48-a56993d52a2a" containerID="b4de98ecb9b5e93adca75b5f18b6bcbfad0fd1da73dc21ff675dae93ca039225" exitCode=0 Sep 08 16:39:34.435816 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:34.435438 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7902-0531d4" event={"ID":"6df0364c-8a9a-419f-bd48-a56993d52a2a","Type":"ContainerDied","Data":"b4de98ecb9b5e93adca75b5f18b6bcbfad0fd1da73dc21ff675dae93ca039225"} Sep 08 16:39:35.576615 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:35.576587 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:35.597658 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:35.597623 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7902-0531d4_6df0364c-8a9a-419f-bd48-a56993d52a2a/mt-curl-7902-0531d4/0.log" Sep 08 16:39:35.619310 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:35.619274 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7902-0531d4"] Sep 08 16:39:35.623399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:35.623371 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7902-0531d4"] Sep 08 16:39:35.662939 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:35.662902 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ztc8\" (UniqueName: \"kubernetes.io/projected/6df0364c-8a9a-419f-bd48-a56993d52a2a-kube-api-access-2ztc8\") pod \"6df0364c-8a9a-419f-bd48-a56993d52a2a\" (UID: \"6df0364c-8a9a-419f-bd48-a56993d52a2a\") " Sep 08 16:39:35.664982 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:35.664957 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df0364c-8a9a-419f-bd48-a56993d52a2a-kube-api-access-2ztc8" (OuterVolumeSpecName: "kube-api-access-2ztc8") pod "6df0364c-8a9a-419f-bd48-a56993d52a2a" (UID: "6df0364c-8a9a-419f-bd48-a56993d52a2a"). InnerVolumeSpecName "kube-api-access-2ztc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:39:35.764491 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:35.764405 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztc8\" (UniqueName: \"kubernetes.io/projected/6df0364c-8a9a-419f-bd48-a56993d52a2a-kube-api-access-2ztc8\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:39:36.449613 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:36.449580 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7166e0aff9b65762537eb055635428a3a126b20d1c1c3c7ab55068a2b2d2157" Sep 08 16:39:36.449613 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:36.449581 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:36.459589 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:39:36.459554 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7902-0531d4\" is forbidden: User \"system:node:ip-10-0-139-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.ec2.internal' and this object" podUID="6df0364c-8a9a-419f-bd48-a56993d52a2a" pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:36.937651 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:39:36.937609 2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7902-0531d4\" is forbidden: User \"system:node:ip-10-0-139-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.ec2.internal' and this object" podUID="6df0364c-8a9a-419f-bd48-a56993d52a2a" pod="opendatahub/mt-curl-7902-0531d4" Sep 08 16:39:36.939358 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:39:36.939336 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df0364c-8a9a-419f-bd48-a56993d52a2a" path="/var/lib/kubelet/pods/6df0364c-8a9a-419f-bd48-a56993d52a2a/volumes" Sep 08 16:40:01.317126 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.317088 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck"] Sep 08 16:40:01.317602 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.317519 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df0364c-8a9a-419f-bd48-a56993d52a2a" containerName="mt-curl-7902-0531d4" Sep 08 16:40:01.317602 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.317532 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df0364c-8a9a-419f-bd48-a56993d52a2a" containerName="mt-curl-7902-0531d4" Sep 08 16:40:01.317676 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.317609 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df0364c-8a9a-419f-bd48-a56993d52a2a" containerName="mt-curl-7902-0531d4" Sep 08 16:40:01.320815 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.320790 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.323951 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.323928 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-402a97-openshift-default-dockercfg-8rl22\"" Sep 08 16:40:01.337955 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.337919 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck"] Sep 08 16:40:01.421968 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.421935 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-data\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.421989 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-certs\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.422038 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-socket\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.422084 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-token\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422150 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.422127 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-podinfo\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.422150 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istiod-ca-cert\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.422174 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-envoy\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.422223 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mq5\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-kube-api-access-69mq5\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.422323 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.422263 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-credential-socket\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523257 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523216 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-certs\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523308 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-socket\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523358 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-token\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523392 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-podinfo\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523412 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istiod-ca-cert\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523431 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-envoy\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523721 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523475 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mq5\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-kube-api-access-69mq5\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523721 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523657 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-credential-socket\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523829 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523713 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-certs\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523829 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523744 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-data\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.523829 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.523725 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-socket\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.524091 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.524062 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-credential-socket\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.524091 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.524080 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-data\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.524321 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.524186 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istiod-ca-cert\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.525956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.525923 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-envoy\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.526141 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.526120 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-podinfo\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.541867 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.541839 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-token\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.542011 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.541957 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mq5\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-kube-api-access-69mq5\") pod \"e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.630937 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.630896 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:01.917829 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.917798 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck"] Sep 08 16:40:01.919474 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:40:01.919444 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d16d50_41b9_49bb_ac8a_2af80a5e275b.slice/crio-f9174828968fae69525ef2cd79254c16a4979acaf73b66e186ae16d726c9bcee WatchSource:0}: Error finding container f9174828968fae69525ef2cd79254c16a4979acaf73b66e186ae16d726c9bcee: Status 404 returned error can't find the container with id f9174828968fae69525ef2cd79254c16a4979acaf73b66e186ae16d726c9bcee Sep 08 16:40:01.921796 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.921760 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:01.921873 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.921850 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:01.921908 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:01.921879 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:02.602728 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:02.602694 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" event={"ID":"30d16d50-41b9-49bb-ac8a-2af80a5e275b","Type":"ContainerStarted","Data":"1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969"} Sep 08 16:40:02.603138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:02.602761 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" event={"ID":"30d16d50-41b9-49bb-ac8a-2af80a5e275b","Type":"ContainerStarted","Data":"f9174828968fae69525ef2cd79254c16a4979acaf73b66e186ae16d726c9bcee"} Sep 08 16:40:02.629105 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:02.629048 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" podStartSLOduration=1.629029124 podStartE2EDuration="1.629029124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:02.626506647 +0000 UTC m=+1636.221505427" watchObservedRunningTime="2026-09-08 16:40:02.629029124 +0000 UTC m=+1636.224027884" Sep 08 16:40:02.631390 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:02.631356 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:02.636577 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:02.636552 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:03.608428 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:03.608393 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:03.609368 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:03.609346 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:23.730892 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:23.730855 2555 generic.go:356] "Generic (PLEG): container finished" podID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerID="1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969" exitCode=0 Sep 08 16:40:23.731366 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:23.730923 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" event={"ID":"30d16d50-41b9-49bb-ac8a-2af80a5e275b","Type":"ContainerDied","Data":"1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969"} Sep 08 16:40:23.731366 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:23.731362 2555 scope.go:117] "RemoveContainer" containerID="1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969" Sep 08 16:40:23.732551 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:23.732520 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:23.732629 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:23.732591 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:23.732629 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:23.732623 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:24.740353 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:24.740317 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" event={"ID":"30d16d50-41b9-49bb-ac8a-2af80a5e275b","Type":"ContainerStarted","Data":"e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68"} Sep 08 16:40:25.631051 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:25.631008 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:25.631378 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:25.631343 2555 patch_prober.go:28] interesting pod/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 08 16:40:25.631514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:25.631410 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 08 16:40:26.631844 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:26.631781 2555 patch_prober.go:28] interesting pod/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 08 16:40:26.632324 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:26.631843 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 08 16:40:27.632159 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:27.632120 2555 patch_prober.go:28] interesting pod/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 08 16:40:27.632603 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:27.632176 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 08 16:40:28.631347 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:28.631305 2555 patch_prober.go:28] interesting pod/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 08 16:40:28.631543 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:28.631364 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 08 16:40:29.631511 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:29.631469 2555 patch_prober.go:28] interesting pod/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Sep 08 16:40:29.632049 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:29.631539 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Sep 08 16:40:30.249436 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.249396 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7"] Sep 08 16:40:30.253140 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.253115 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.255845 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.255822 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e7d9b1-openshift-default-dockercfg-64225\"" Sep 08 16:40:30.278886 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.278854 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7"] Sep 08 16:40:30.309209 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309165 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-podinfo\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309209 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309212 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-data\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309321 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istiod-ca-cert\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309346 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgndz\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-kube-api-access-bgndz\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309367 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-socket\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309390 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-certs\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309478 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-token\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309597 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309586 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-credential-socket\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.309848 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.309619 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-envoy\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411181 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411141 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istiod-ca-cert\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411181 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgndz\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-kube-api-access-bgndz\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411206 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-socket\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411225 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-certs\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411280 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-token\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411333 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-credential-socket\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411358 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-envoy\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411414 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-podinfo\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411484 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411441 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-data\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411697 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-socket\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411753 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-certs\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.411856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411795 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-data\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.412027 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411925 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-credential-socket\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.412027 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.411958 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istiod-ca-cert\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.413813 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.413788 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-envoy\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.413929 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.413885 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-podinfo\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.419261 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.419221 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-token\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.419350 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.419312 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgndz\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-kube-api-access-bgndz\") pod \"e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.563390 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.563290 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:30.636768 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.636737 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:30.637209 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.637133 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:30.637993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.637976 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:40:30.847188 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.847158 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7"] Sep 08 16:40:30.848674 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:40:30.848647 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5e2786_5af1_4dc6_9107_ab1f8fabdd6b.slice/crio-9192492d2156b4e453e9e2126f6609a2d0192003cf80e8bcce72afe8f1db5e2b WatchSource:0}: Error finding container 9192492d2156b4e453e9e2126f6609a2d0192003cf80e8bcce72afe8f1db5e2b: Status 404 returned error can't find the container with id 9192492d2156b4e453e9e2126f6609a2d0192003cf80e8bcce72afe8f1db5e2b Sep 08 16:40:30.851219 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.851183 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:30.851342 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.851302 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:30.851413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:30.851350 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:31.796493 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:31.796455 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" event={"ID":"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b","Type":"ContainerStarted","Data":"3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa"} Sep 08 16:40:31.797042 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:31.796507 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" event={"ID":"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b","Type":"ContainerStarted","Data":"9192492d2156b4e453e9e2126f6609a2d0192003cf80e8bcce72afe8f1db5e2b"} Sep 08 16:40:31.858192 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:31.858132 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" podStartSLOduration=1.8581155950000001 podStartE2EDuration="1.858115595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:31.856532001 +0000 UTC m=+1665.451530779" watchObservedRunningTime="2026-09-08 16:40:31.858115595 +0000 UTC m=+1665.453114353" Sep 08 16:40:32.564055 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:32.563993 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:32.569408 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:32.569376 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:32.802516 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:32.802480 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:32.803459 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:32.803436 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:35.011840 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.011796 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr"] Sep 08 16:40:35.015841 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.015816 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.018689 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.018662 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e7d9b1-serving-cert\"" Sep 08 16:40:35.028196 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.028170 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr"] Sep 08 16:40:35.055981 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.055936 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c812e60a-4dcb-48f5-a7c4-f94be37cde82-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.055981 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.055986 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-metrics-certs\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.056229 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.056003 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56x5f\" (UniqueName: \"kubernetes.io/projected/c812e60a-4dcb-48f5-a7c4-f94be37cde82-kube-api-access-56x5f\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.056229 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.056139 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.157732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.157685 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c812e60a-4dcb-48f5-a7c4-f94be37cde82-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.157732 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.157736 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-metrics-certs\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.158010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.157754 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56x5f\" (UniqueName: \"kubernetes.io/projected/c812e60a-4dcb-48f5-a7c4-f94be37cde82-kube-api-access-56x5f\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.158010 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.157829 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.158010 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:40:35.157996 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-serving-cert: secret "maas-api-e2e-api-b-e7d9b1-serving-cert" not found Sep 08 16:40:35.158172 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:40:35.158070 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls podName:c812e60a-4dcb-48f5-a7c4-f94be37cde82 nodeName:}" failed. No retries permitted until 2026-09-08 16:40:35.658050701 +0000 UTC m=+1669.253049442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls") pod "maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" (UID: "c812e60a-4dcb-48f5-a7c4-f94be37cde82") : secret "maas-api-e2e-api-b-e7d9b1-serving-cert" not found Sep 08 16:40:35.158504 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.158481 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c812e60a-4dcb-48f5-a7c4-f94be37cde82-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.160274 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.160231 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-metrics-certs\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.182345 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.182311 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56x5f\" (UniqueName: \"kubernetes.io/projected/c812e60a-4dcb-48f5-a7c4-f94be37cde82-kube-api-access-56x5f\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.664514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.664471 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.667158 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.667129 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls\") pod \"maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:35.927940 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:35.927852 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:36.210222 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:36.210191 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr"] Sep 08 16:40:36.211349 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:40:36.211322 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc812e60a_4dcb_48f5_a7c4_f94be37cde82.slice/crio-2761fdb2415637f72891e51d24df2a3127914d5b656ebb85ee6d2d02266d3c80 WatchSource:0}: Error finding container 2761fdb2415637f72891e51d24df2a3127914d5b656ebb85ee6d2d02266d3c80: Status 404 returned error can't find the container with id 2761fdb2415637f72891e51d24df2a3127914d5b656ebb85ee6d2d02266d3c80 Sep 08 16:40:36.212851 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:36.212831 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:40:36.833338 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:36.833220 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" event={"ID":"c812e60a-4dcb-48f5-a7c4-f94be37cde82","Type":"ContainerStarted","Data":"e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc"} Sep 08 16:40:36.833506 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:36.833404 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:36.833506 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:36.833435 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" event={"ID":"c812e60a-4dcb-48f5-a7c4-f94be37cde82","Type":"ContainerStarted","Data":"2761fdb2415637f72891e51d24df2a3127914d5b656ebb85ee6d2d02266d3c80"} Sep 08 16:40:36.867693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:36.867638 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" podStartSLOduration=2.525794404 podStartE2EDuration="2.867619424s" totalImagesPullingTime="341.82502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:36.865536001 +0000 UTC m=+1670.460534789" watchObservedRunningTime="2026-09-08 16:40:36.867619424 +0000 UTC m=+1670.462618182" Sep 08 16:40:42.845583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:42.845546 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:40:52.933460 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:52.933425 2555 generic.go:356] "Generic (PLEG): container finished" podID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerID="3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa" exitCode=0 Sep 08 16:40:52.936742 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:52.936714 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" event={"ID":"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b","Type":"ContainerDied","Data":"3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa"} Sep 08 16:40:52.937120 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:52.937106 2555 scope.go:117] "RemoveContainer" containerID="3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa" Sep 08 16:40:52.938305 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:52.938278 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:52.938360 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:52.938346 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:52.938396 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:52.938380 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:40:53.942529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:53.942485 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" event={"ID":"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b","Type":"ContainerStarted","Data":"03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43"} Sep 08 16:40:54.564104 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:54.564057 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:54.564355 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:54.564195 2555 patch_prober.go:28] interesting pod/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 08 16:40:54.564355 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:54.564258 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 08 16:40:55.563870 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:55.563818 2555 patch_prober.go:28] interesting pod/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 08 16:40:55.564422 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:55.563898 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 08 16:40:56.564513 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:56.564465 2555 patch_prober.go:28] interesting pod/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 08 16:40:56.565036 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:56.564554 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 08 16:40:57.564099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:57.564040 2555 patch_prober.go:28] interesting pod/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 08 16:40:57.564465 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:57.564433 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 08 16:40:58.564268 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:58.564200 2555 patch_prober.go:28] interesting pod/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 08 16:40:58.564900 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:58.564308 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 08 16:40:59.568381 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:59.568346 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:59.568872 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:59.568736 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:40:59.569549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:40:59.569532 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:41:24.406656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:24.406570 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck"] Sep 08 16:41:24.407213 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:24.406803 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" containerID="cri-o://e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68" gracePeriod=30 Sep 08 16:41:29.654175 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.654146 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:41:29.834161 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834068 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-envoy\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834161 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834135 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-credential-socket\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834197 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-podinfo\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834227 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-data\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834285 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-socket\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834325 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-token\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834390 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69mq5\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-kube-api-access-69mq5\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834417 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-certs\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834452 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istiod-ca-cert\") pod \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\" (UID: \"30d16d50-41b9-49bb-ac8a-2af80a5e275b\") " Sep 08 16:41:29.834563 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834537 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834592 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834622 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-data" (OuterVolumeSpecName: "istio-data") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834791 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834861 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834881 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834896 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834910 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.835002 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.834909 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:41:29.836571 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.836537 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:29.836683 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.836593 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:41:29.836683 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.836596 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-kube-api-access-69mq5" (OuterVolumeSpecName: "kube-api-access-69mq5") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "kube-api-access-69mq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:41:29.836918 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.836898 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-token" (OuterVolumeSpecName: "istio-token") pod "30d16d50-41b9-49bb-ac8a-2af80a5e275b" (UID: "30d16d50-41b9-49bb-ac8a-2af80a5e275b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:41:29.935811 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.935765 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69mq5\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-kube-api-access-69mq5\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.935811 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.935803 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.935811 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.935812 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.935811 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.935821 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:29.935811 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:29.935828 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d16d50-41b9-49bb-ac8a-2af80a5e275b-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:30.179500 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.179465 2555 generic.go:356] "Generic (PLEG): container finished" podID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerID="e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68" exitCode=0 Sep 08 16:41:30.179697 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.179515 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" Sep 08 16:41:30.179697 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.179551 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" event={"ID":"30d16d50-41b9-49bb-ac8a-2af80a5e275b","Type":"ContainerDied","Data":"e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68"} Sep 08 16:41:30.179697 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.179604 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck" event={"ID":"30d16d50-41b9-49bb-ac8a-2af80a5e275b","Type":"ContainerDied","Data":"f9174828968fae69525ef2cd79254c16a4979acaf73b66e186ae16d726c9bcee"} Sep 08 16:41:30.179697 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.179627 2555 scope.go:117] "RemoveContainer" containerID="e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68" Sep 08 16:41:30.190169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.190147 2555 scope.go:117] "RemoveContainer" containerID="1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969" Sep 08 16:41:30.199725 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.199700 2555 scope.go:117] "RemoveContainer" containerID="e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68" Sep 08 16:41:30.200113 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:41:30.200055 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68\": container with ID starting with e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68 not found: ID does not exist" containerID="e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68" Sep 08 16:41:30.200294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.200100 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68"} err="failed to get container status \"e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68\": rpc error: code = NotFound desc = could not find container \"e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68\": container with ID starting with e0f4a01889c5da3c534ac90b821bd0ae318ace9da0ed023c3eed25fc3b79aa68 not found: ID does not exist" Sep 08 16:41:30.200294 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.200131 2555 scope.go:117] "RemoveContainer" containerID="1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969" Sep 08 16:41:30.200918 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:41:30.200883 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969\": container with ID starting with 1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969 not found: ID does not exist" containerID="1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969" Sep 08 16:41:30.201006 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.200917 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969"} err="failed to get container status \"1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969\": rpc error: code = NotFound desc = could not find container \"1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969\": container with ID starting with 1880a06cf1e11cc533f743751a110e52f9f790778ea2fc2222ce5edb5b7f4969 not found: ID does not exist" Sep 08 16:41:30.203930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.203896 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck"] Sep 08 16:41:30.207361 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.207338 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-402a97-openshift-default-c8474b5db-7g9ck"] Sep 08 16:41:30.937925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:30.937887 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" path="/var/lib/kubelet/pods/30d16d50-41b9-49bb-ac8a-2af80a5e275b/volumes" Sep 08 16:41:40.314454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.314418 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr"] Sep 08 16:41:40.314897 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.314657 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" podUID="c812e60a-4dcb-48f5-a7c4-f94be37cde82" containerName="maas-api" containerID="cri-o://e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc" gracePeriod=30 Sep 08 16:41:40.566465 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.566390 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:41:40.639640 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.639601 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls\") pod \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " Sep 08 16:41:40.639825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.639666 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c812e60a-4dcb-48f5-a7c4-f94be37cde82-openshift-service-ca\") pod \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " Sep 08 16:41:40.639825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.639699 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56x5f\" (UniqueName: \"kubernetes.io/projected/c812e60a-4dcb-48f5-a7c4-f94be37cde82-kube-api-access-56x5f\") pod \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " Sep 08 16:41:40.639825 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.639777 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-metrics-certs\") pod \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\" (UID: \"c812e60a-4dcb-48f5-a7c4-f94be37cde82\") " Sep 08 16:41:40.640131 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.640102 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c812e60a-4dcb-48f5-a7c4-f94be37cde82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c812e60a-4dcb-48f5-a7c4-f94be37cde82" (UID: "c812e60a-4dcb-48f5-a7c4-f94be37cde82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:41:40.641780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.641756 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c812e60a-4dcb-48f5-a7c4-f94be37cde82" (UID: "c812e60a-4dcb-48f5-a7c4-f94be37cde82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:41:40.641903 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.641880 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c812e60a-4dcb-48f5-a7c4-f94be37cde82-kube-api-access-56x5f" (OuterVolumeSpecName: "kube-api-access-56x5f") pod "c812e60a-4dcb-48f5-a7c4-f94be37cde82" (UID: "c812e60a-4dcb-48f5-a7c4-f94be37cde82"). InnerVolumeSpecName "kube-api-access-56x5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:41:40.641968 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.641916 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c812e60a-4dcb-48f5-a7c4-f94be37cde82" (UID: "c812e60a-4dcb-48f5-a7c4-f94be37cde82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:41:40.741134 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.741097 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:40.741134 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.741133 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c812e60a-4dcb-48f5-a7c4-f94be37cde82-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:40.741134 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.741143 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56x5f\" (UniqueName: \"kubernetes.io/projected/c812e60a-4dcb-48f5-a7c4-f94be37cde82-kube-api-access-56x5f\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:40.741424 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:40.741152 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c812e60a-4dcb-48f5-a7c4-f94be37cde82-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:41.248939 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.248851 2555 generic.go:356] "Generic (PLEG): container finished" podID="c812e60a-4dcb-48f5-a7c4-f94be37cde82" containerID="e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc" exitCode=0 Sep 08 16:41:41.248939 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.248897 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" Sep 08 16:41:41.249130 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.248934 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" event={"ID":"c812e60a-4dcb-48f5-a7c4-f94be37cde82","Type":"ContainerDied","Data":"e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc"} Sep 08 16:41:41.249130 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.248989 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr" event={"ID":"c812e60a-4dcb-48f5-a7c4-f94be37cde82","Type":"ContainerDied","Data":"2761fdb2415637f72891e51d24df2a3127914d5b656ebb85ee6d2d02266d3c80"} Sep 08 16:41:41.249130 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.249013 2555 scope.go:117] "RemoveContainer" containerID="e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc" Sep 08 16:41:41.262211 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.262189 2555 scope.go:117] "RemoveContainer" containerID="e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc" Sep 08 16:41:41.262561 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:41:41.262537 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc\": container with ID starting with e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc not found: ID does not exist" containerID="e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc" Sep 08 16:41:41.262623 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.262571 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc"} err="failed to get container status \"e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc\": rpc error: code = NotFound desc = could not find container \"e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc\": container with ID starting with e8643349130bce9fbe4ac47e0c1227aae3c1bfa07d9c3ab755bca0fe279784bc not found: ID does not exist" Sep 08 16:41:41.272859 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.272824 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr"] Sep 08 16:41:41.275290 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:41.275265 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7d9b1-bf5778d48-xfkhr"] Sep 08 16:41:42.937990 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:42.937953 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c812e60a-4dcb-48f5-a7c4-f94be37cde82" path="/var/lib/kubelet/pods/c812e60a-4dcb-48f5-a7c4-f94be37cde82/volumes" Sep 08 16:41:47.380658 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:47.380560 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7"] Sep 08 16:41:47.381116 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:47.380840 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" containerID="cri-o://03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43" gracePeriod=30 Sep 08 16:41:52.635234 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.635205 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:41:52.759914 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.759813 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgndz\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-kube-api-access-bgndz\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.759914 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.759868 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-socket\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.759914 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.759897 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-data\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.759914 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.759920 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-certs\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.759943 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-token\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760017 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istiod-ca-cert\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760062 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-envoy\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760098 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-podinfo\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760145 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-credential-socket\") pod \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\" (UID: \"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b\") " Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760188 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-data" (OuterVolumeSpecName: "istio-data") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760219 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:52.760319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760230 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:52.760769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760521 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:52.760769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760554 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.760769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760579 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.760769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760594 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.760769 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.760695 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:41:52.762255 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.762216 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:41:52.762358 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.762255 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:41:52.762539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.762525 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-kube-api-access-bgndz" (OuterVolumeSpecName: "kube-api-access-bgndz") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "kube-api-access-bgndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:41:52.762583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.762553 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" (UID: "de5e2786-5af1-4dc6-9107-ab1f8fabdd6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:41:52.862107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.862062 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgndz\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-kube-api-access-bgndz\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.862107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.862100 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.862107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.862110 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.862107 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.862118 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.862415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.862125 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:52.862415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:52.862133 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:41:53.342797 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.342752 2555 generic.go:356] "Generic (PLEG): container finished" podID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerID="03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43" exitCode=0 Sep 08 16:41:53.342976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.342808 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" Sep 08 16:41:53.342976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.342833 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" event={"ID":"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b","Type":"ContainerDied","Data":"03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43"} Sep 08 16:41:53.342976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.342894 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7" event={"ID":"de5e2786-5af1-4dc6-9107-ab1f8fabdd6b","Type":"ContainerDied","Data":"9192492d2156b4e453e9e2126f6609a2d0192003cf80e8bcce72afe8f1db5e2b"} Sep 08 16:41:53.342976 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.342923 2555 scope.go:117] "RemoveContainer" containerID="03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43" Sep 08 16:41:53.352958 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.352937 2555 scope.go:117] "RemoveContainer" containerID="3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa" Sep 08 16:41:53.361792 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.361760 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7"] Sep 08 16:41:53.362679 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.362649 2555 scope.go:117] "RemoveContainer" containerID="03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43" Sep 08 16:41:53.363033 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:41:53.363014 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43\": container with ID starting with 03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43 not found: ID does not exist" containerID="03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43" Sep 08 16:41:53.363085 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.363042 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43"} err="failed to get container status \"03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43\": rpc error: code = NotFound desc = could not find container \"03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43\": container with ID starting with 03b7b1ace8565f5971f8f9ea98e34199edc90cf77119ed26052f6f650e04cf43 not found: ID does not exist" Sep 08 16:41:53.363085 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.363061 2555 scope.go:117] "RemoveContainer" containerID="3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa" Sep 08 16:41:53.363327 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:41:53.363308 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa\": container with ID starting with 3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa not found: ID does not exist" containerID="3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa" Sep 08 16:41:53.363380 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.363335 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa"} err="failed to get container status \"3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa\": rpc error: code = NotFound desc = could not find container \"3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa\": container with ID starting with 3be9f23c1ea26199521cee1ca8044d576cb2b9f6d13ae12b5992db201b0f04aa not found: ID does not exist" Sep 08 16:41:53.368369 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.368338 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e7d9b1-openshift-default-8495cc49fd-bfms7"] Sep 08 16:41:53.852172 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852137 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5"] Sep 08 16:41:53.852605 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852591 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" Sep 08 16:41:53.852605 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852606 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" Sep 08 16:41:53.852684 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852624 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" Sep 08 16:41:53.852684 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852630 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" Sep 08 16:41:53.852684 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852647 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c812e60a-4dcb-48f5-a7c4-f94be37cde82" containerName="maas-api" Sep 08 16:41:53.852684 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852654 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c812e60a-4dcb-48f5-a7c4-f94be37cde82" containerName="maas-api" Sep 08 16:41:53.852837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852716 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" Sep 08 16:41:53.852837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852726 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" Sep 08 16:41:53.852837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852733 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" Sep 08 16:41:53.852837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852739 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" Sep 08 16:41:53.852837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.852747 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c812e60a-4dcb-48f5-a7c4-f94be37cde82" containerName="maas-api" Sep 08 16:41:53.857500 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.857472 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.860228 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.860203 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-352142-openshift-default-dockercfg-sgldg\"" Sep 08 16:41:53.872611 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.872581 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5"] Sep 08 16:41:53.973377 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973332 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-certs\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973388 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-envoy\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973471 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-data\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973516 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istiod-ca-cert\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973669 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973565 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-credential-socket\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973669 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973616 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-token\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973669 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973646 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c254m\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-kube-api-access-c254m\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973766 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973721 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-podinfo\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:53.973766 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:53.973741 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-socket\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.074648 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.074601 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-token\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.074850 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.074670 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c254m\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-kube-api-access-c254m\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.074850 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.074721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-podinfo\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.074850 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.074766 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-socket\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.074850 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.074846 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-certs\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075079 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.074877 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-envoy\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075079 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.074979 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-data\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075079 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.075045 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istiod-ca-cert\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075230 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.075084 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-credential-socket\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075377 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.075349 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-certs\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075472 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.075442 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-socket\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075740 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.075711 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-credential-socket\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075860 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.075834 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-data\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.075935 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.075878 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istiod-ca-cert\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.077349 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.077322 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-envoy\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.077572 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.077548 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-podinfo\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.083503 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.083469 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-token\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.083652 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.083630 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c254m\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-kube-api-access-c254m\") pod \"e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.168419 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.168323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:54.447474 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.447437 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5"] Sep 08 16:41:54.448893 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:41:54.448863 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5b38cf_abb4_4579_8ca3_21a7685ee0aa.slice/crio-5330bab5c23753b6536d2148876a047be625b8af930b090b57e7cbb7d8e5b648 WatchSource:0}: Error finding container 5330bab5c23753b6536d2148876a047be625b8af930b090b57e7cbb7d8e5b648: Status 404 returned error can't find the container with id 5330bab5c23753b6536d2148876a047be625b8af930b090b57e7cbb7d8e5b648 Sep 08 16:41:54.451054 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.451019 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:41:54.451143 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.451104 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:41:54.451216 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.451150 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:41:54.937857 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:54.937806 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" path="/var/lib/kubelet/pods/de5e2786-5af1-4dc6-9107-ab1f8fabdd6b/volumes" Sep 08 16:41:55.358119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:55.358080 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" event={"ID":"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa","Type":"ContainerStarted","Data":"c94718b65c340bc3a9c27803c42388d795916020eb9ec044fa103b36af9b1fd1"} Sep 08 16:41:55.358373 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:55.358133 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" event={"ID":"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa","Type":"ContainerStarted","Data":"5330bab5c23753b6536d2148876a047be625b8af930b090b57e7cbb7d8e5b648"} Sep 08 16:41:55.380437 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:55.380372 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" podStartSLOduration=2.380355555 podStartE2EDuration="2.380355555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:55.377000111 +0000 UTC m=+1748.971998895" watchObservedRunningTime="2026-09-08 16:41:55.380355555 +0000 UTC m=+1748.975354314" Sep 08 16:41:56.168918 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:56.168874 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:56.174397 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:56.174365 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:56.365308 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:56.365272 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:56.366284 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:56.366239 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:41:58.517852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.514623 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn"] Sep 08 16:41:58.517852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.515647 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" Sep 08 16:41:58.517852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.515670 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5e2786-5af1-4dc6-9107-ab1f8fabdd6b" containerName="istio-proxy" Sep 08 16:41:58.517852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.515702 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" Sep 08 16:41:58.517852 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.515711 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d16d50-41b9-49bb-ac8a-2af80a5e275b" containerName="istio-proxy" Sep 08 16:41:58.523120 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.523086 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.526413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.526353 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-352142-serving-cert\"" Sep 08 16:41:58.530205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.530169 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn"] Sep 08 16:41:58.623123 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.623072 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.623343 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.623196 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.623343 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.623226 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt4ft\" (UniqueName: \"kubernetes.io/projected/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-kube-api-access-dt4ft\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.623343 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.623298 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-metrics-certs\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.724338 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.724291 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.724549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.724390 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.724549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.724417 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4ft\" (UniqueName: \"kubernetes.io/projected/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-kube-api-access-dt4ft\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.724549 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.724471 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-metrics-certs\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.724729 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:41:58.724544 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-serving-cert: secret "maas-api-e2e-ipp-a-352142-serving-cert" not found Sep 08 16:41:58.724729 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:41:58.724643 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls podName:80bc94bf-0cbf-4ee6-9e95-be089e318cc6 nodeName:}" failed. No retries permitted until 2026-09-08 16:41:59.224619495 +0000 UTC m=+1752.819618257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls") pod "maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" (UID: "80bc94bf-0cbf-4ee6-9e95-be089e318cc6") : secret "maas-api-e2e-ipp-a-352142-serving-cert" not found Sep 08 16:41:58.725089 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.725066 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.727271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.727218 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-metrics-certs\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:58.735113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:58.735076 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4ft\" (UniqueName: \"kubernetes.io/projected/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-kube-api-access-dt4ft\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:59.230514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:59.230470 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:59.232931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:59.232903 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:59.439319 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:59.439275 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:41:59.720439 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:41:59.720407 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn"] Sep 08 16:41:59.721841 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:41:59.721811 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bc94bf_0cbf_4ee6_9e95_be089e318cc6.slice/crio-255c43524f39a20e6f32b74149b923ece185c01b087864964aea148032e4930b WatchSource:0}: Error finding container 255c43524f39a20e6f32b74149b923ece185c01b087864964aea148032e4930b: Status 404 returned error can't find the container with id 255c43524f39a20e6f32b74149b923ece185c01b087864964aea148032e4930b Sep 08 16:42:00.389818 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:00.389779 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" event={"ID":"80bc94bf-0cbf-4ee6-9e95-be089e318cc6","Type":"ContainerStarted","Data":"dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9"} Sep 08 16:42:00.390021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:00.389831 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" event={"ID":"80bc94bf-0cbf-4ee6-9e95-be089e318cc6","Type":"ContainerStarted","Data":"255c43524f39a20e6f32b74149b923ece185c01b087864964aea148032e4930b"} Sep 08 16:42:00.390021 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:00.389912 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:42:00.408956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:00.408892 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" podStartSLOduration=1.925289475 podStartE2EDuration="2.408869882s" totalImagesPullingTime="483.580407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:00.405874058 +0000 UTC m=+1754.000872837" watchObservedRunningTime="2026-09-08 16:42:00.408869882 +0000 UTC m=+1754.003868668" Sep 08 16:42:06.401498 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:06.401466 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:42:15.497527 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:15.497438 2555 generic.go:356] "Generic (PLEG): container finished" podID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerID="c94718b65c340bc3a9c27803c42388d795916020eb9ec044fa103b36af9b1fd1" exitCode=0 Sep 08 16:42:15.497527 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:15.497508 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" event={"ID":"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa","Type":"ContainerDied","Data":"c94718b65c340bc3a9c27803c42388d795916020eb9ec044fa103b36af9b1fd1"} Sep 08 16:42:15.497993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:15.497915 2555 scope.go:117] "RemoveContainer" containerID="c94718b65c340bc3a9c27803c42388d795916020eb9ec044fa103b36af9b1fd1" Sep 08 16:42:15.499158 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:15.499122 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:42:15.499214 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:15.499203 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:42:15.499278 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:15.499236 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:42:16.116989 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.116953 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r"] Sep 08 16:42:16.117441 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.117394 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" podUID="2deda381-a97c-4ba5-b940-49c20908d299" containerName="maas-api" containerID="cri-o://39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38" gracePeriod=30 Sep 08 16:42:16.375767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.375697 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:42:16.507357 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.507319 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" event={"ID":"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa","Type":"ContainerStarted","Data":"861f13212ab098c762641a94be212ec048fbb2400cafc18ae93fc7cc982baa44"} Sep 08 16:42:16.509417 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.509387 2555 generic.go:356] "Generic (PLEG): container finished" podID="2deda381-a97c-4ba5-b940-49c20908d299" containerID="39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38" exitCode=0 Sep 08 16:42:16.509548 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.509442 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" Sep 08 16:42:16.509548 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.509445 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" event={"ID":"2deda381-a97c-4ba5-b940-49c20908d299","Type":"ContainerDied","Data":"39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38"} Sep 08 16:42:16.509631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.509588 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r" event={"ID":"2deda381-a97c-4ba5-b940-49c20908d299","Type":"ContainerDied","Data":"cb962d3c774ca30aced262ca12fde1c2a91be758aeb4b3e9953e5e94949f98ca"} Sep 08 16:42:16.509631 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.509622 2555 scope.go:117] "RemoveContainer" containerID="39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38" Sep 08 16:42:16.513383 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.513345 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deda381-a97c-4ba5-b940-49c20908d299-openshift-service-ca\") pod \"2deda381-a97c-4ba5-b940-49c20908d299\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " Sep 08 16:42:16.513529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.513400 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls\") pod \"2deda381-a97c-4ba5-b940-49c20908d299\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " Sep 08 16:42:16.513529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.513523 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-metrics-certs\") pod \"2deda381-a97c-4ba5-b940-49c20908d299\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " Sep 08 16:42:16.513662 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.513617 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr7l5\" (UniqueName: \"kubernetes.io/projected/2deda381-a97c-4ba5-b940-49c20908d299-kube-api-access-nr7l5\") pod \"2deda381-a97c-4ba5-b940-49c20908d299\" (UID: \"2deda381-a97c-4ba5-b940-49c20908d299\") " Sep 08 16:42:16.515913 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.515522 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deda381-a97c-4ba5-b940-49c20908d299-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2deda381-a97c-4ba5-b940-49c20908d299" (UID: "2deda381-a97c-4ba5-b940-49c20908d299"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:42:16.516893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.516855 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deda381-a97c-4ba5-b940-49c20908d299-kube-api-access-nr7l5" (OuterVolumeSpecName: "kube-api-access-nr7l5") pod "2deda381-a97c-4ba5-b940-49c20908d299" (UID: "2deda381-a97c-4ba5-b940-49c20908d299"). InnerVolumeSpecName "kube-api-access-nr7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:42:16.530856 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.530827 2555 scope.go:117] "RemoveContainer" containerID="39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38" Sep 08 16:42:16.531008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.530816 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2deda381-a97c-4ba5-b940-49c20908d299" (UID: "2deda381-a97c-4ba5-b940-49c20908d299"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:42:16.531008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.530856 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2deda381-a97c-4ba5-b940-49c20908d299" (UID: "2deda381-a97c-4ba5-b940-49c20908d299"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:42:16.531440 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:42:16.531371 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38\": container with ID starting with 39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38 not found: ID does not exist" containerID="39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38" Sep 08 16:42:16.531440 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.531413 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38"} err="failed to get container status \"39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38\": rpc error: code = NotFound desc = could not find container \"39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38\": container with ID starting with 39ce71a88cfd762a594ced027c40368d9413a55ce406f08c1803d506484c9e38 not found: ID does not exist" Sep 08 16:42:16.614705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.614662 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7l5\" (UniqueName: \"kubernetes.io/projected/2deda381-a97c-4ba5-b940-49c20908d299-kube-api-access-nr7l5\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:16.614705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.614704 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deda381-a97c-4ba5-b940-49c20908d299-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:16.614993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.614720 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:16.614993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.614733 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2deda381-a97c-4ba5-b940-49c20908d299-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:16.832306 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.832273 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r"] Sep 08 16:42:16.834897 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.834866 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-8ded33-68b5fdb6bd-qqp9r"] Sep 08 16:42:16.937583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:16.937549 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2deda381-a97c-4ba5-b940-49c20908d299" path="/var/lib/kubelet/pods/2deda381-a97c-4ba5-b940-49c20908d299/volumes" Sep 08 16:42:17.168716 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:17.168677 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:42:17.169028 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:17.168999 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 08 16:42:17.169088 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:17.169067 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 08 16:42:18.169115 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:18.169077 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 08 16:42:18.169579 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:18.169137 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 08 16:42:19.168757 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:19.168711 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 08 16:42:19.168993 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:19.168783 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 08 16:42:20.168757 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:20.168719 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 08 16:42:20.169167 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:20.168787 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 08 16:42:21.169479 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:21.169438 2555 patch_prober.go:28] interesting pod/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 08 16:42:21.169893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:21.169504 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 08 16:42:22.299125 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:22.299081 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:42:22.299679 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:22.299513 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:42:22.301779 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:22.301756 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:42:23.252054 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:23.252011 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx"] Sep 08 16:42:23.252276 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:23.252222 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" containerID="cri-o://482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4" gracePeriod=30 Sep 08 16:42:28.517909 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.517835 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:42:28.595556 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.595515 2555 generic.go:356] "Generic (PLEG): container finished" podID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerID="482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4" exitCode=0 Sep 08 16:42:28.595743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.595570 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" Sep 08 16:42:28.595743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.595570 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" event={"ID":"54af582b-54c5-40ac-85ac-d1bd199c00fc","Type":"ContainerDied","Data":"482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4"} Sep 08 16:42:28.595743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.595627 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx" event={"ID":"54af582b-54c5-40ac-85ac-d1bd199c00fc","Type":"ContainerDied","Data":"f92d3b3cd7b4102406c470b0b58e4f72c045ae2553353080c92b793f332cc6d7"} Sep 08 16:42:28.595743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.595651 2555 scope.go:117] "RemoveContainer" containerID="482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4" Sep 08 16:42:28.605122 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.605100 2555 scope.go:117] "RemoveContainer" containerID="556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f" Sep 08 16:42:28.614473 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.614452 2555 scope.go:117] "RemoveContainer" containerID="482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4" Sep 08 16:42:28.614822 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:42:28.614801 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4\": container with ID starting with 482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4 not found: ID does not exist" containerID="482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4" Sep 08 16:42:28.614878 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.614832 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4"} err="failed to get container status \"482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4\": rpc error: code = NotFound desc = could not find container \"482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4\": container with ID starting with 482c1689266f9d36003617f9ee21d23aec6a2cab59241b529febb36c9265e7b4 not found: ID does not exist" Sep 08 16:42:28.614878 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.614853 2555 scope.go:117] "RemoveContainer" containerID="556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f" Sep 08 16:42:28.615144 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:42:28.615125 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f\": container with ID starting with 556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f not found: ID does not exist" containerID="556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f" Sep 08 16:42:28.615187 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.615152 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f"} err="failed to get container status \"556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f\": rpc error: code = NotFound desc = could not find container \"556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f\": container with ID starting with 556fd79a7549b9ce784010ba606cb4bb95b05b6f5be6e59c61c1fa285d91a17f not found: ID does not exist" Sep 08 16:42:28.627525 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627497 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-token\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.627695 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627538 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-podinfo\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.627695 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627557 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54af582b-54c5-40ac-85ac-d1bd199c00fc-istiod-ca-cert\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.627810 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627738 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-credential-socket\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.627866 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627813 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4nq8\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-kube-api-access-b4nq8\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.627866 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627861 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-certs\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.627977 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627904 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-envoy\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.627977 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627965 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-data\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.628060 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.627980 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:42:28.628060 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628021 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-socket\") pod \"54af582b-54c5-40ac-85ac-d1bd199c00fc\" (UID: \"54af582b-54c5-40ac-85ac-d1bd199c00fc\") " Sep 08 16:42:28.628194 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628156 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:42:28.628369 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628329 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:42:28.628455 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628403 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:42:28.628517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628458 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.628517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628481 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.628517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628494 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.628647 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.628532 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54af582b-54c5-40ac-85ac-d1bd199c00fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:42:28.629923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.629901 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:42:28.629995 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.629923 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:42:28.630369 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.630351 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:42:28.630369 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.630355 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-kube-api-access-b4nq8" (OuterVolumeSpecName: "kube-api-access-b4nq8") pod "54af582b-54c5-40ac-85ac-d1bd199c00fc" (UID: "54af582b-54c5-40ac-85ac-d1bd199c00fc"). InnerVolumeSpecName "kube-api-access-b4nq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:42:28.729458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.729414 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4nq8\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-kube-api-access-b4nq8\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.729458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.729454 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.729458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.729464 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54af582b-54c5-40ac-85ac-d1bd199c00fc-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.729696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.729473 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.729696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.729486 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54af582b-54c5-40ac-85ac-d1bd199c00fc-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.729696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.729494 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54af582b-54c5-40ac-85ac-d1bd199c00fc-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:42:28.919185 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.919144 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx"] Sep 08 16:42:28.925893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.925854 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-8ded33-openshift-default-588979f6d6-l72dx"] Sep 08 16:42:28.937723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:28.937690 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" path="/var/lib/kubelet/pods/54af582b-54c5-40ac-85ac-d1bd199c00fc/volumes" Sep 08 16:42:47.119365 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:47.119330 2555 scope.go:117] "RemoveContainer" containerID="cbc167cbc809aeb425e18fbd5e8100812f0ffc0426b7c3e046e20a4482332b45" Sep 08 16:42:47.129499 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:42:47.129473 2555 scope.go:117] "RemoveContainer" containerID="30e897674b79b10b46bde52b0fc9dbab052e885a944949c0e15b525b6fba4caa" Sep 08 16:43:32.281773 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.281739 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj"] Sep 08 16:43:32.282286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282197 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deda381-a97c-4ba5-b940-49c20908d299" containerName="maas-api" Sep 08 16:43:32.282286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282211 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deda381-a97c-4ba5-b940-49c20908d299" containerName="maas-api" Sep 08 16:43:32.282286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282230 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" Sep 08 16:43:32.282286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282236 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" Sep 08 16:43:32.282286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282265 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" Sep 08 16:43:32.282286 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282273 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" Sep 08 16:43:32.282488 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282350 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" Sep 08 16:43:32.282488 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.282363 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deda381-a97c-4ba5-b940-49c20908d299" containerName="maas-api" Sep 08 16:43:32.284583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.284560 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.287474 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.287446 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8dca27-openshift-default-dockercfg-m8qbl\"" Sep 08 16:43:32.298205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.298176 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj"] Sep 08 16:43:32.458641 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458596 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-socket\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.458837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458704 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpvg\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-kube-api-access-5kpvg\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.458837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458730 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-envoy\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.458837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458773 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-credential-socket\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.458837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458807 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-token\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.458837 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458836 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-data\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.459016 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458857 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-certs\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.459016 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458908 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.459016 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.458952 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-podinfo\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.559925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.559825 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpvg\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-kube-api-access-5kpvg\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.559925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.559871 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-envoy\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.559993 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-credential-socket\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560062 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-token\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560112 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-data\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560342 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560311 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-certs\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560385 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560391 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-credential-socket\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560451 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560421 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-podinfo\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560676 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560464 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-socket\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560676 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560494 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-data\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560676 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560632 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-certs\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.560848 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560811 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-socket\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.561012 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.560991 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.562238 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.562219 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-envoy\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.563144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.563121 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-podinfo\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.573443 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.573408 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpvg\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-kube-api-access-5kpvg\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.574898 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.574868 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-token\") pod \"e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.596843 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.596805 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:32.873663 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.873632 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj"] Sep 08 16:43:32.875287 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:43:32.875258 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode984f0e0_76b1_4fc9_9b45_5eb6477acf38.slice/crio-56d884355171098249cc294829fd52edc8a57abf0ac6e57f87108f8f15d5be55 WatchSource:0}: Error finding container 56d884355171098249cc294829fd52edc8a57abf0ac6e57f87108f8f15d5be55: Status 404 returned error can't find the container with id 56d884355171098249cc294829fd52edc8a57abf0ac6e57f87108f8f15d5be55 Sep 08 16:43:32.877429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.877397 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:43:32.877492 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.877464 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:43:32.877533 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.877493 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:43:32.992594 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.992544 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" event={"ID":"e984f0e0-76b1-4fc9-9b45-5eb6477acf38","Type":"ContainerStarted","Data":"c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c"} Sep 08 16:43:32.992768 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:32.992618 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" event={"ID":"e984f0e0-76b1-4fc9-9b45-5eb6477acf38","Type":"ContainerStarted","Data":"56d884355171098249cc294829fd52edc8a57abf0ac6e57f87108f8f15d5be55"} Sep 08 16:43:33.014310 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:33.014221 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podStartSLOduration=1.014205783 podStartE2EDuration="1.014205783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:33.012876888 +0000 UTC m=+1846.607875673" watchObservedRunningTime="2026-09-08 16:43:33.014205783 +0000 UTC m=+1846.609204542" Sep 08 16:43:33.596999 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:33.596954 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:33.598471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:33.598444 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 08 16:43:33.598543 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:33.598502 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 08 16:43:34.602516 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:34.602482 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:35.005448 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:35.005358 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:35.006336 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:35.006313 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:47.170684 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:47.170651 2555 scope.go:117] "RemoveContainer" containerID="35a12a9b0e8794b847f40ecd3e8076ef9949ffef2c8482e371f5c05dea5f83ac" Sep 08 16:43:56.138687 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:56.138649 2555 generic.go:356] "Generic (PLEG): container finished" podID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerID="c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c" exitCode=0 Sep 08 16:43:56.139271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:56.138725 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" event={"ID":"e984f0e0-76b1-4fc9-9b45-5eb6477acf38","Type":"ContainerDied","Data":"c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c"} Sep 08 16:43:56.139361 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:56.139288 2555 scope.go:117] "RemoveContainer" containerID="c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c" Sep 08 16:43:56.140458 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:56.140427 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:43:56.140567 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:56.140514 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:43:56.140632 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:56.140566 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:43:57.152766 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:57.152711 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" event={"ID":"e984f0e0-76b1-4fc9-9b45-5eb6477acf38","Type":"ContainerStarted","Data":"328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434"} Sep 08 16:43:57.597333 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:57.597281 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:43:57.597510 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:57.597462 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 08 16:43:57.597571 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:57.597514 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 08 16:43:58.597787 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:58.597743 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 08 16:43:58.598218 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:58.597819 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 08 16:43:59.597551 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:59.597507 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 08 16:43:59.597749 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:43:59.597571 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 08 16:44:00.522685 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.522651 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62"] Sep 08 16:44:00.523156 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.523142 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="54af582b-54c5-40ac-85ac-d1bd199c00fc" containerName="istio-proxy" Sep 08 16:44:00.525317 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.525297 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" Sep 08 16:44:00.527842 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.527820 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\"" Sep 08 16:44:00.534115 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.534081 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62"] Sep 08 16:44:00.598004 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.597967 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 08 16:44:00.598169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.598044 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 08 16:44:00.665777 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.665732 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7004ed9-177c-4d8f-b379-6f74c96e4fbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62\" (UID: \"f7004ed9-177c-4d8f-b379-6f74c96e4fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" Sep 08 16:44:00.766780 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.766736 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7004ed9-177c-4d8f-b379-6f74c96e4fbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62\" (UID: \"f7004ed9-177c-4d8f-b379-6f74c96e4fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" Sep 08 16:44:00.767497 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.767471 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7004ed9-177c-4d8f-b379-6f74c96e4fbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62\" (UID: \"f7004ed9-177c-4d8f-b379-6f74c96e4fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" Sep 08 16:44:00.836880 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:00.836761 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" Sep 08 16:44:01.128981 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:01.128948 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62"] Sep 08 16:44:01.130460 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:44:01.130433 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7004ed9_177c_4d8f_b379_6f74c96e4fbf.slice/crio-105cfc36a6e3c1e2bedd5f4b9878d8376506cc38e11b485504d1251dfd3d2b3e WatchSource:0}: Error finding container 105cfc36a6e3c1e2bedd5f4b9878d8376506cc38e11b485504d1251dfd3d2b3e: Status 404 returned error can't find the container with id 105cfc36a6e3c1e2bedd5f4b9878d8376506cc38e11b485504d1251dfd3d2b3e Sep 08 16:44:01.184303 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:01.184263 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" event={"ID":"f7004ed9-177c-4d8f-b379-6f74c96e4fbf","Type":"ContainerStarted","Data":"105cfc36a6e3c1e2bedd5f4b9878d8376506cc38e11b485504d1251dfd3d2b3e"} Sep 08 16:44:01.598149 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:01.598109 2555 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Sep 08 16:44:01.598602 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:01.598175 2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Sep 08 16:44:02.617925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:02.617892 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:44:02.618464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:02.618349 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:44:02.619330 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:02.619311 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:44:05.212695 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:05.212655 2555 generic.go:356] "Generic (PLEG): container finished" podID="f7004ed9-177c-4d8f-b379-6f74c96e4fbf" containerID="8781bd4c2c1469d1a9c0bc842064e19d3e99bf45b43e99dc110de0568b583455" exitCode=0 Sep 08 16:44:05.213159 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:05.212734 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" event={"ID":"f7004ed9-177c-4d8f-b379-6f74c96e4fbf","Type":"ContainerDied","Data":"8781bd4c2c1469d1a9c0bc842064e19d3e99bf45b43e99dc110de0568b583455"} Sep 08 16:44:06.356959 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:06.356930 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" Sep 08 16:44:06.422533 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:06.422493 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7004ed9-177c-4d8f-b379-6f74c96e4fbf-maas-api-service-ca\") pod \"f7004ed9-177c-4d8f-b379-6f74c96e4fbf\" (UID: \"f7004ed9-177c-4d8f-b379-6f74c96e4fbf\") " Sep 08 16:44:06.422882 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:06.422859 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7004ed9-177c-4d8f-b379-6f74c96e4fbf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7004ed9-177c-4d8f-b379-6f74c96e4fbf" (UID: "f7004ed9-177c-4d8f-b379-6f74c96e4fbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:44:06.524066 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:06.523976 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7004ed9-177c-4d8f-b379-6f74c96e4fbf-maas-api-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:07.225634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:07.225549 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" Sep 08 16:44:07.225634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:07.225567 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62" event={"ID":"f7004ed9-177c-4d8f-b379-6f74c96e4fbf","Type":"ContainerDied","Data":"105cfc36a6e3c1e2bedd5f4b9878d8376506cc38e11b485504d1251dfd3d2b3e"} Sep 08 16:44:07.225634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:07.225615 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105cfc36a6e3c1e2bedd5f4b9878d8376506cc38e11b485504d1251dfd3d2b3e" Sep 08 16:44:17.316656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:17.316567 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62"] Sep 08 16:44:17.321343 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:17.321311 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8dca27-96f62"] Sep 08 16:44:18.936926 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:18.936889 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7004ed9-177c-4d8f-b379-6f74c96e4fbf" path="/var/lib/kubelet/pods/f7004ed9-177c-4d8f-b379-6f74c96e4fbf/volumes" Sep 08 16:44:23.852534 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:23.852496 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj"] Sep 08 16:44:23.853144 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:23.852733 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" containerID="cri-o://328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434" gracePeriod=30 Sep 08 16:44:29.105676 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.105648 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:44:29.244065 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.243957 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-data\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244068 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istiod-ca-cert\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244102 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-podinfo\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244129 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-certs\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244159 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-envoy\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244269 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244216 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kpvg\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-kube-api-access-5kpvg\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244323 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-socket\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244357 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-data" (OuterVolumeSpecName: "istio-data") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:44:29.244546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244382 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-token\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244387 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:44:29.244546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244421 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-credential-socket\") pod \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\" (UID: \"e984f0e0-76b1-4fc9-9b45-5eb6477acf38\") " Sep 08 16:44:29.244793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244566 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:44:29.244793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244586 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:44:29.244793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244736 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.244793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244744 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:44:29.244793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244754 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.244793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244770 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.244793 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.244783 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.246583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.246554 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-kube-api-access-5kpvg" (OuterVolumeSpecName: "kube-api-access-5kpvg") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "kube-api-access-5kpvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:44:29.246583 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.246574 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-token" (OuterVolumeSpecName: "istio-token") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:44:29.246775 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.246757 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:44:29.246871 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.246848 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e984f0e0-76b1-4fc9-9b45-5eb6477acf38" (UID: "e984f0e0-76b1-4fc9-9b45-5eb6477acf38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:44:29.345930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.345890 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.345930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.345924 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.345930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.345935 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.345930 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.345944 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.346211 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.345953 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kpvg\" (UniqueName: \"kubernetes.io/projected/e984f0e0-76b1-4fc9-9b45-5eb6477acf38-kube-api-access-5kpvg\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:29.367568 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.367524 2555 generic.go:356] "Generic (PLEG): container finished" podID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerID="328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434" exitCode=0 Sep 08 16:44:29.367767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.367581 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" Sep 08 16:44:29.367767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.367574 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" event={"ID":"e984f0e0-76b1-4fc9-9b45-5eb6477acf38","Type":"ContainerDied","Data":"328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434"} Sep 08 16:44:29.367767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.367718 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj" event={"ID":"e984f0e0-76b1-4fc9-9b45-5eb6477acf38","Type":"ContainerDied","Data":"56d884355171098249cc294829fd52edc8a57abf0ac6e57f87108f8f15d5be55"} Sep 08 16:44:29.367767 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.367759 2555 scope.go:117] "RemoveContainer" containerID="328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434" Sep 08 16:44:29.378171 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.378151 2555 scope.go:117] "RemoveContainer" containerID="c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c" Sep 08 16:44:29.387358 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.387335 2555 scope.go:117] "RemoveContainer" containerID="328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434" Sep 08 16:44:29.387671 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:44:29.387653 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434\": container with ID starting with 328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434 not found: ID does not exist" containerID="328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434" Sep 08 16:44:29.387751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.387680 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434"} err="failed to get container status \"328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434\": rpc error: code = NotFound desc = could not find container \"328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434\": container with ID starting with 328b9d329e87028daaf7d1e33f74b55930405dba1db5c6993fb04d6e2f5d8434 not found: ID does not exist" Sep 08 16:44:29.387751 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.387697 2555 scope.go:117] "RemoveContainer" containerID="c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c" Sep 08 16:44:29.387953 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:44:29.387932 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c\": container with ID starting with c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c not found: ID does not exist" containerID="c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c" Sep 08 16:44:29.388000 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.387957 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c"} err="failed to get container status \"c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c\": rpc error: code = NotFound desc = could not find container \"c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c\": container with ID starting with c8e1024ef8694c3fa1bebba32f4541446d7db500f9315deb5cfa6b4efaec654c not found: ID does not exist" Sep 08 16:44:29.395964 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.395936 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj"] Sep 08 16:44:29.398417 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:29.398394 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8dca27-openshift-default-7b98b86694-f72nj"] Sep 08 16:44:30.937331 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:30.937297 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" path="/var/lib/kubelet/pods/e984f0e0-76b1-4fc9-9b45-5eb6477acf38/volumes" Sep 08 16:44:42.914718 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:42.914672 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn"] Sep 08 16:44:42.915236 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:42.914936 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" podUID="80bc94bf-0cbf-4ee6-9e95-be089e318cc6" containerName="maas-api" containerID="cri-o://dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9" gracePeriod=30 Sep 08 16:44:43.170958 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.170889 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:44:43.287092 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.287042 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls\") pod \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " Sep 08 16:44:43.287332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.287148 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt4ft\" (UniqueName: \"kubernetes.io/projected/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-kube-api-access-dt4ft\") pod \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " Sep 08 16:44:43.287332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.287221 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-metrics-certs\") pod \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " Sep 08 16:44:43.287332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.287296 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-openshift-service-ca\") pod \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\" (UID: \"80bc94bf-0cbf-4ee6-9e95-be089e318cc6\") " Sep 08 16:44:43.287685 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.287656 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80bc94bf-0cbf-4ee6-9e95-be089e318cc6" (UID: "80bc94bf-0cbf-4ee6-9e95-be089e318cc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:44:43.289423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.289392 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80bc94bf-0cbf-4ee6-9e95-be089e318cc6" (UID: "80bc94bf-0cbf-4ee6-9e95-be089e318cc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:44:43.289423 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.289418 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80bc94bf-0cbf-4ee6-9e95-be089e318cc6" (UID: "80bc94bf-0cbf-4ee6-9e95-be089e318cc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:44:43.289576 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.289441 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-kube-api-access-dt4ft" (OuterVolumeSpecName: "kube-api-access-dt4ft") pod "80bc94bf-0cbf-4ee6-9e95-be089e318cc6" (UID: "80bc94bf-0cbf-4ee6-9e95-be089e318cc6"). InnerVolumeSpecName "kube-api-access-dt4ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:44:43.388136 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.388090 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt4ft\" (UniqueName: \"kubernetes.io/projected/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-kube-api-access-dt4ft\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:43.388136 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.388128 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:43.388136 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.388142 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:43.388425 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.388153 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80bc94bf-0cbf-4ee6-9e95-be089e318cc6-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:44:43.454692 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.454601 2555 generic.go:356] "Generic (PLEG): container finished" podID="80bc94bf-0cbf-4ee6-9e95-be089e318cc6" containerID="dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9" exitCode=0 Sep 08 16:44:43.454692 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.454645 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" Sep 08 16:44:43.454953 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.454689 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" event={"ID":"80bc94bf-0cbf-4ee6-9e95-be089e318cc6","Type":"ContainerDied","Data":"dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9"} Sep 08 16:44:43.454953 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.454751 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn" event={"ID":"80bc94bf-0cbf-4ee6-9e95-be089e318cc6","Type":"ContainerDied","Data":"255c43524f39a20e6f32b74149b923ece185c01b087864964aea148032e4930b"} Sep 08 16:44:43.454953 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.454780 2555 scope.go:117] "RemoveContainer" containerID="dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9" Sep 08 16:44:43.464830 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.464809 2555 scope.go:117] "RemoveContainer" containerID="dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9" Sep 08 16:44:43.465140 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:44:43.465121 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9\": container with ID starting with dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9 not found: ID does not exist" containerID="dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9" Sep 08 16:44:43.465203 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.465153 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9"} err="failed to get container status \"dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9\": rpc error: code = NotFound desc = could not find container \"dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9\": container with ID starting with dabb764e27519e6eef166aef245feff8bcf7ca32f4ce7a33acb32355fbb5a8b9 not found: ID does not exist" Sep 08 16:44:43.477553 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.477518 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn"] Sep 08 16:44:43.480190 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:43.480168 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-352142-855dd7475c-z7fhn"] Sep 08 16:44:44.938035 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:44:44.937949 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80bc94bf-0cbf-4ee6-9e95-be089e318cc6" path="/var/lib/kubelet/pods/80bc94bf-0cbf-4ee6-9e95-be089e318cc6/volumes" Sep 08 16:45:00.161366 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161327 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh"] Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161795 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7004ed9-177c-4d8f-b379-6f74c96e4fbf" containerName="revoke-keys" Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161810 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7004ed9-177c-4d8f-b379-6f74c96e4fbf" containerName="revoke-keys" Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161825 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161831 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161845 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80bc94bf-0cbf-4ee6-9e95-be089e318cc6" containerName="maas-api" Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161850 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="80bc94bf-0cbf-4ee6-9e95-be089e318cc6" containerName="maas-api" Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161856 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" Sep 08 16:45:00.161888 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161861 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" Sep 08 16:45:00.162178 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161925 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7004ed9-177c-4d8f-b379-6f74c96e4fbf" containerName="revoke-keys" Sep 08 16:45:00.162178 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161934 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="80bc94bf-0cbf-4ee6-9e95-be089e318cc6" containerName="maas-api" Sep 08 16:45:00.162178 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161941 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" Sep 08 16:45:00.162178 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.161949 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e984f0e0-76b1-4fc9-9b45-5eb6477acf38" containerName="istio-proxy" Sep 08 16:45:00.165462 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.165444 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh" Sep 08 16:45:00.177390 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.177362 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\"" Sep 08 16:45:00.179620 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.179595 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc"] Sep 08 16:45:00.183558 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.183540 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc" Sep 08 16:45:00.185693 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.185670 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh" Sep 08 16:45:00.192237 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.191771 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh"] Sep 08 16:45:00.196103 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.196080 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc" Sep 08 16:45:00.208349 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.208221 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc"] Sep 08 16:45:00.484099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.484066 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh"] Sep 08 16:45:00.485732 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:45:00.485706 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9534e3_cd3e_47b6_8516_b02373f8b6f0.slice/crio-8e3bf9582fe1ece7268968880438814bd456e89a9c12f9b680fdf6944ac25f7e WatchSource:0}: Error finding container 8e3bf9582fe1ece7268968880438814bd456e89a9c12f9b680fdf6944ac25f7e: Status 404 returned error can't find the container with id 8e3bf9582fe1ece7268968880438814bd456e89a9c12f9b680fdf6944ac25f7e Sep 08 16:45:00.494943 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.494916 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc"] Sep 08 16:45:00.496273 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:45:00.496231 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e452b7_6f92_4c84_8a33_5505fa0f6e4e.slice/crio-27e4f421d1946049424f3f07a15d71eec43916fcdead23f6d7f005d3e3e7fae1 WatchSource:0}: Error finding container 27e4f421d1946049424f3f07a15d71eec43916fcdead23f6d7f005d3e3e7fae1: Status 404 returned error can't find the container with id 27e4f421d1946049424f3f07a15d71eec43916fcdead23f6d7f005d3e3e7fae1 Sep 08 16:45:00.560061 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.560019 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh" event={"ID":"ca9534e3-cd3e-47b6-8516-b02373f8b6f0","Type":"ContainerStarted","Data":"8e3bf9582fe1ece7268968880438814bd456e89a9c12f9b680fdf6944ac25f7e"} Sep 08 16:45:00.561351 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:00.561326 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc" event={"ID":"83e452b7-6f92-4c84-8a33-5505fa0f6e4e","Type":"ContainerStarted","Data":"27e4f421d1946049424f3f07a15d71eec43916fcdead23f6d7f005d3e3e7fae1"} Sep 08 16:45:01.568413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:01.568375 2555 generic.go:356] "Generic (PLEG): container finished" podID="ca9534e3-cd3e-47b6-8516-b02373f8b6f0" containerID="a59e927690d582354d2ca3a2f6fb23ebbb5ab4a7337ad4f5a96a1e7add6b61e8" exitCode=0 Sep 08 16:45:01.568861 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:01.568458 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh" event={"ID":"ca9534e3-cd3e-47b6-8516-b02373f8b6f0","Type":"ContainerDied","Data":"a59e927690d582354d2ca3a2f6fb23ebbb5ab4a7337ad4f5a96a1e7add6b61e8"} Sep 08 16:45:01.570159 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:01.570133 2555 generic.go:356] "Generic (PLEG): container finished" podID="83e452b7-6f92-4c84-8a33-5505fa0f6e4e" containerID="9b64f4c55d04b26765e1f31092cc3066a45c805bae7a7cf2def1a98e02f6e0a9" exitCode=0 Sep 08 16:45:01.570309 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:01.570210 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc" event={"ID":"83e452b7-6f92-4c84-8a33-5505fa0f6e4e","Type":"ContainerDied","Data":"9b64f4c55d04b26765e1f31092cc3066a45c805bae7a7cf2def1a98e02f6e0a9"} Sep 08 16:45:02.729850 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:02.729825 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc" Sep 08 16:45:02.733373 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:02.733345 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh" Sep 08 16:45:03.586080 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:03.586052 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh" Sep 08 16:45:03.586270 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:03.586052 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814765-67zhh" event={"ID":"ca9534e3-cd3e-47b6-8516-b02373f8b6f0","Type":"ContainerDied","Data":"8e3bf9582fe1ece7268968880438814bd456e89a9c12f9b680fdf6944ac25f7e"} Sep 08 16:45:03.586270 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:03.586173 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e3bf9582fe1ece7268968880438814bd456e89a9c12f9b680fdf6944ac25f7e" Sep 08 16:45:03.587430 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:03.587409 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc" Sep 08 16:45:03.587545 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:03.587452 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc" event={"ID":"83e452b7-6f92-4c84-8a33-5505fa0f6e4e","Type":"ContainerDied","Data":"27e4f421d1946049424f3f07a15d71eec43916fcdead23f6d7f005d3e3e7fae1"} Sep 08 16:45:03.587545 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:03.587490 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e4f421d1946049424f3f07a15d71eec43916fcdead23f6d7f005d3e3e7fae1" Sep 08 16:45:37.652417 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:37.652379 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5"] Sep 08 16:45:37.652958 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:37.652682 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" containerID="cri-o://861f13212ab098c762641a94be212ec048fbb2400cafc18ae93fc7cc982baa44" gracePeriod=30 Sep 08 16:45:38.009066 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.008972 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt"] Sep 08 16:45:38.009640 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.009615 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e452b7-6f92-4c84-8a33-5505fa0f6e4e" containerName="cleanup" Sep 08 16:45:38.009733 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.009643 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e452b7-6f92-4c84-8a33-5505fa0f6e4e" containerName="cleanup" Sep 08 16:45:38.009733 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.009668 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9534e3-cd3e-47b6-8516-b02373f8b6f0" containerName="cleanup" Sep 08 16:45:38.009733 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.009677 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9534e3-cd3e-47b6-8516-b02373f8b6f0" containerName="cleanup" Sep 08 16:45:38.009858 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.009796 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9534e3-cd3e-47b6-8516-b02373f8b6f0" containerName="cleanup" Sep 08 16:45:38.009858 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.009812 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e452b7-6f92-4c84-8a33-5505fa0f6e4e" containerName="cleanup" Sep 08 16:45:38.013555 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.013532 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" Sep 08 16:45:38.016411 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.016374 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\"" Sep 08 16:45:38.022634 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.022276 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt"] Sep 08 16:45:38.095975 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.095938 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07620007-2da5-48bf-9f5c-52569cd8437c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt\" (UID: \"07620007-2da5-48bf-9f5c-52569cd8437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" Sep 08 16:45:38.197327 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.197289 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07620007-2da5-48bf-9f5c-52569cd8437c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt\" (UID: \"07620007-2da5-48bf-9f5c-52569cd8437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" Sep 08 16:45:38.197931 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.197909 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07620007-2da5-48bf-9f5c-52569cd8437c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt\" (UID: \"07620007-2da5-48bf-9f5c-52569cd8437c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" Sep 08 16:45:38.327400 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.327298 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" Sep 08 16:45:38.603938 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.603910 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt"] Sep 08 16:45:38.605846 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:45:38.605821 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07620007_2da5_48bf_9f5c_52569cd8437c.slice/crio-01b9ac340dc6698fad47bc411f4b14a3245f757c95eb00e16db453706979a077 WatchSource:0}: Error finding container 01b9ac340dc6698fad47bc411f4b14a3245f757c95eb00e16db453706979a077: Status 404 returned error can't find the container with id 01b9ac340dc6698fad47bc411f4b14a3245f757c95eb00e16db453706979a077 Sep 08 16:45:38.796069 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:38.796032 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" event={"ID":"07620007-2da5-48bf-9f5c-52569cd8437c","Type":"ContainerStarted","Data":"01b9ac340dc6698fad47bc411f4b14a3245f757c95eb00e16db453706979a077"} Sep 08 16:45:39.803731 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:39.803697 2555 generic.go:356] "Generic (PLEG): container finished" podID="07620007-2da5-48bf-9f5c-52569cd8437c" containerID="b7419c16754935ce601672cf31f437e363bcc409017ae13490cae12d205ce99d" exitCode=0 Sep 08 16:45:39.804168 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:39.803783 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" event={"ID":"07620007-2da5-48bf-9f5c-52569cd8437c","Type":"ContainerDied","Data":"b7419c16754935ce601672cf31f437e363bcc409017ae13490cae12d205ce99d"} Sep 08 16:45:40.952123 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:40.952096 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" Sep 08 16:45:41.024146 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:41.024092 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07620007-2da5-48bf-9f5c-52569cd8437c-maas-api-service-ca\") pod \"07620007-2da5-48bf-9f5c-52569cd8437c\" (UID: \"07620007-2da5-48bf-9f5c-52569cd8437c\") " Sep 08 16:45:41.024523 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:41.024494 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07620007-2da5-48bf-9f5c-52569cd8437c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07620007-2da5-48bf-9f5c-52569cd8437c" (UID: "07620007-2da5-48bf-9f5c-52569cd8437c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:45:41.125798 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:41.125762 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07620007-2da5-48bf-9f5c-52569cd8437c-maas-api-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:41.816519 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:41.816486 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" Sep 08 16:45:41.816729 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:41.816531 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt" event={"ID":"07620007-2da5-48bf-9f5c-52569cd8437c","Type":"ContainerDied","Data":"01b9ac340dc6698fad47bc411f4b14a3245f757c95eb00e16db453706979a077"} Sep 08 16:45:41.816729 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:41.816580 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b9ac340dc6698fad47bc411f4b14a3245f757c95eb00e16db453706979a077" Sep 08 16:45:42.826775 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:42.826736 2555 generic.go:356] "Generic (PLEG): container finished" podID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerID="861f13212ab098c762641a94be212ec048fbb2400cafc18ae93fc7cc982baa44" exitCode=0 Sep 08 16:45:42.827179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:42.826808 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" event={"ID":"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa","Type":"ContainerDied","Data":"861f13212ab098c762641a94be212ec048fbb2400cafc18ae93fc7cc982baa44"} Sep 08 16:45:42.827179 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:42.826874 2555 scope.go:117] "RemoveContainer" containerID="c94718b65c340bc3a9c27803c42388d795916020eb9ec044fa103b36af9b1fd1" Sep 08 16:45:42.918687 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:42.918612 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:45:43.042407 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042370 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c254m\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-kube-api-access-c254m\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042432 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-envoy\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042457 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-socket\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042509 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istiod-ca-cert\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042543 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-credential-socket\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042588 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042573 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-podinfo\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042867 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042602 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-certs\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042867 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042654 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-data\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042867 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042693 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-token\") pod \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\" (UID: \"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa\") " Sep 08 16:45:43.042867 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042816 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:45:43.042867 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042838 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:45:43.043138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.042924 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:45:43.043138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.043019 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:45:43.043138 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.043120 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.043318 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.043141 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.043318 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.043157 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.043318 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.043158 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:45:43.043318 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.043172 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.044735 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.044714 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:45:43.044814 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.044763 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:45:43.045030 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.045016 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-kube-api-access-c254m" (OuterVolumeSpecName: "kube-api-access-c254m") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "kube-api-access-c254m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:45:43.045131 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.045110 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" (UID: "fc5b38cf-abb4-4579-8ca3-21a7685ee0aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:45:43.143991 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.143944 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.143991 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.143981 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c254m\" (UniqueName: \"kubernetes.io/projected/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-kube-api-access-c254m\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.143991 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.143996 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.144277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.144008 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.144277 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.144020 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:45:43.834470 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.834437 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" Sep 08 16:45:43.834906 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.834477 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5" event={"ID":"fc5b38cf-abb4-4579-8ca3-21a7685ee0aa","Type":"ContainerDied","Data":"5330bab5c23753b6536d2148876a047be625b8af930b090b57e7cbb7d8e5b648"} Sep 08 16:45:43.834906 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.834539 2555 scope.go:117] "RemoveContainer" containerID="861f13212ab098c762641a94be212ec048fbb2400cafc18ae93fc7cc982baa44" Sep 08 16:45:43.857305 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.857230 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5"] Sep 08 16:45:43.860937 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:43.860908 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-352142-openshift-default-66c497fbb4-n84q5"] Sep 08 16:45:44.937429 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:44.937345 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" path="/var/lib/kubelet/pods/fc5b38cf-abb4-4579-8ca3-21a7685ee0aa/volumes" Sep 08 16:45:47.212991 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:47.212955 2555 scope.go:117] "RemoveContainer" containerID="b4de98ecb9b5e93adca75b5f18b6bcbfad0fd1da73dc21ff675dae93ca039225" Sep 08 16:45:48.710311 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:48.710271 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt"] Sep 08 16:45:48.712639 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:48.712601 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-995b22-9hhwt"] Sep 08 16:45:48.937313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:45:48.937275 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07620007-2da5-48bf-9f5c-52569cd8437c" path="/var/lib/kubelet/pods/07620007-2da5-48bf-9f5c-52569cd8437c/volumes" Sep 08 16:46:06.219893 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:06.219842 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc"] Sep 08 16:46:06.222206 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:06.222182 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bce20b-29814765-dxwtc"] Sep 08 16:46:06.938786 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:06.938751 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e452b7-6f92-4c84-8a33-5505fa0f6e4e" path="/var/lib/kubelet/pods/83e452b7-6f92-4c84-8a33-5505fa0f6e4e/volumes" Sep 08 16:46:30.797518 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:30.797475 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg"] Sep 08 16:46:30.798199 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:30.797690 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" containerID="cri-o://34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46" gracePeriod=30 Sep 08 16:46:33.334778 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.334727 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587797bcc9-bw8rs"] Sep 08 16:46:33.335307 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.335005 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-587797bcc9-bw8rs" podUID="0f252198-3fcd-4324-9cce-42145a51914b" containerName="manager" containerID="cri-o://bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876" gracePeriod=30 Sep 08 16:46:33.583434 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.583403 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:46:33.708713 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.708674 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-metrics-certs\") pod \"0f252198-3fcd-4324-9cce-42145a51914b\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " Sep 08 16:46:33.708921 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.708769 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-webhook-cert\") pod \"0f252198-3fcd-4324-9cce-42145a51914b\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " Sep 08 16:46:33.708921 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.708807 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzlhn\" (UniqueName: \"kubernetes.io/projected/0f252198-3fcd-4324-9cce-42145a51914b-kube-api-access-hzlhn\") pod \"0f252198-3fcd-4324-9cce-42145a51914b\" (UID: \"0f252198-3fcd-4324-9cce-42145a51914b\") " Sep 08 16:46:33.710921 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.710883 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f252198-3fcd-4324-9cce-42145a51914b-kube-api-access-hzlhn" (OuterVolumeSpecName: "kube-api-access-hzlhn") pod "0f252198-3fcd-4324-9cce-42145a51914b" (UID: "0f252198-3fcd-4324-9cce-42145a51914b"). InnerVolumeSpecName "kube-api-access-hzlhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:46:33.710921 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.710908 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f252198-3fcd-4324-9cce-42145a51914b" (UID: "0f252198-3fcd-4324-9cce-42145a51914b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:46:33.711089 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.711010 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0f252198-3fcd-4324-9cce-42145a51914b" (UID: "0f252198-3fcd-4324-9cce-42145a51914b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:46:33.809718 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.809677 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-webhook-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:33.809718 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.809711 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzlhn\" (UniqueName: \"kubernetes.io/projected/0f252198-3fcd-4324-9cce-42145a51914b-kube-api-access-hzlhn\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:33.809718 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:33.809721 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f252198-3fcd-4324-9cce-42145a51914b-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:34.127813 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.127775 2555 generic.go:356] "Generic (PLEG): container finished" podID="0f252198-3fcd-4324-9cce-42145a51914b" containerID="bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876" exitCode=0 Sep 08 16:46:34.128008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.127830 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-bw8rs" Sep 08 16:46:34.128008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.127853 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-bw8rs" event={"ID":"0f252198-3fcd-4324-9cce-42145a51914b","Type":"ContainerDied","Data":"bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876"} Sep 08 16:46:34.128008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.127908 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-bw8rs" event={"ID":"0f252198-3fcd-4324-9cce-42145a51914b","Type":"ContainerDied","Data":"33458e722a8944d5a4b0c44d4010e2040556fb3e30fc5f660941ea58f28590b6"} Sep 08 16:46:34.128008 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.127930 2555 scope.go:117] "RemoveContainer" containerID="bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876" Sep 08 16:46:34.138559 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.138534 2555 scope.go:117] "RemoveContainer" containerID="bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876" Sep 08 16:46:34.138888 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:46:34.138871 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876\": container with ID starting with bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876 not found: ID does not exist" containerID="bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876" Sep 08 16:46:34.138946 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.138896 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876"} err="failed to get container status \"bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876\": rpc error: code = NotFound desc = could not find container \"bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876\": container with ID starting with bc03af7d4c7dd36ac456082bb0869c1f533b5f5e21028b6ac99dcc4323f7b876 not found: ID does not exist" Sep 08 16:46:34.151681 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.151648 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587797bcc9-bw8rs"] Sep 08 16:46:34.155413 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.155381 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-587797bcc9-bw8rs"] Sep 08 16:46:34.937953 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:34.937905 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f252198-3fcd-4324-9cce-42145a51914b" path="/var/lib/kubelet/pods/0f252198-3fcd-4324-9cce-42145a51914b/volumes" Sep 08 16:46:36.053514 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.053483 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:46:36.131448 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131414 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-data\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.131610 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131466 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c25e40-7029-4c25-9e4d-4508b1363ce5-istiod-ca-cert\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.131610 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131504 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-socket\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.131610 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131533 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-podinfo\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.131610 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131559 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-278pq\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-kube-api-access-278pq\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.131796 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131731 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-credential-socket\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.131855 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131758 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-data" (OuterVolumeSpecName: "istio-data") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:46:36.131855 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131830 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:46:36.131855 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131837 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-token\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.132018 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131884 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-certs\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.132018 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131911 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-envoy\") pod \"50c25e40-7029-4c25-9e4d-4508b1363ce5\" (UID: \"50c25e40-7029-4c25-9e4d-4508b1363ce5\") " Sep 08 16:46:36.132018 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.131997 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:46:36.132184 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.132044 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c25e40-7029-4c25-9e4d-4508b1363ce5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:46:36.132184 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.132144 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:46:36.132346 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.132312 2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c25e40-7029-4c25-9e4d-4508b1363ce5-istiod-ca-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.132415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.132346 2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.132415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.132367 2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-credential-socket\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.132415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.132380 2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-workload-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.132415 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.132394 2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-data\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.133941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.133914 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-token" (OuterVolumeSpecName: "istio-token") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:46:36.133941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.133929 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-kube-api-access-278pq" (OuterVolumeSpecName: "kube-api-access-278pq") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "kube-api-access-278pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:46:36.134094 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.133936 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:46:36.134094 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.134021 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50c25e40-7029-4c25-9e4d-4508b1363ce5" (UID: "50c25e40-7029-4c25-9e4d-4508b1363ce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 16:46:36.144280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.144223 2555 generic.go:356] "Generic (PLEG): container finished" podID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerID="34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46" exitCode=0 Sep 08 16:46:36.144435 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.144281 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" event={"ID":"50c25e40-7029-4c25-9e4d-4508b1363ce5","Type":"ContainerDied","Data":"34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46"} Sep 08 16:46:36.144435 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.144353 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" event={"ID":"50c25e40-7029-4c25-9e4d-4508b1363ce5","Type":"ContainerDied","Data":"46d4c074480b8710e5d6d5e9840b5f47ab53dd56afd07965441c3d26d7a59d4a"} Sep 08 16:46:36.144435 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.144295 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg" Sep 08 16:46:36.144435 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.144385 2555 scope.go:117] "RemoveContainer" containerID="34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46" Sep 08 16:46:36.156656 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.156630 2555 scope.go:117] "RemoveContainer" containerID="1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c" Sep 08 16:46:36.169383 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.169344 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg"] Sep 08 16:46:36.174119 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.174094 2555 scope.go:117] "RemoveContainer" containerID="34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46" Sep 08 16:46:36.174204 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.174132 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e22de8-openshift-default-6c66dff5c8-tp8pg"] Sep 08 16:46:36.174525 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:46:36.174506 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46\": container with ID starting with 34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46 not found: ID does not exist" containerID="34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46" Sep 08 16:46:36.174608 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.174537 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46"} err="failed to get container status \"34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46\": rpc error: code = NotFound desc = could not find container \"34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46\": container with ID starting with 34285af5c98c3d0b383893fe9ef5e736bb88f23508a8de9b30be6c98f5885f46 not found: ID does not exist" Sep 08 16:46:36.174608 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.174561 2555 scope.go:117] "RemoveContainer" containerID="1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c" Sep 08 16:46:36.174875 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:46:36.174857 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c\": container with ID starting with 1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c not found: ID does not exist" containerID="1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c" Sep 08 16:46:36.174925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.174881 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c"} err="failed to get container status \"1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c\": rpc error: code = NotFound desc = could not find container \"1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c\": container with ID starting with 1a555854c5300b01a96b6e99443b4d8f224d94171b60bfa9630eda72e4b8533c not found: ID does not exist" Sep 08 16:46:36.233731 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.233684 2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-podinfo\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.233731 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.233723 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278pq\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-kube-api-access-278pq\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.233731 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.233735 2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-token\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.233951 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.233743 2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c25e40-7029-4c25-9e4d-4508b1363ce5-istio-envoy\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:46:36.937475 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:46:36.937436 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" path="/var/lib/kubelet/pods/50c25e40-7029-4c25-9e4d-4508b1363ce5/volumes" Sep 08 16:47:34.957485 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.957438 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587797bcc9-rn447"] Sep 08 16:47:34.958317 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958296 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f252198-3fcd-4324-9cce-42145a51914b" containerName="manager" Sep 08 16:47:34.958381 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958321 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f252198-3fcd-4324-9cce-42145a51914b" containerName="manager" Sep 08 16:47:34.958381 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958333 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" Sep 08 16:47:34.958381 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958342 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" Sep 08 16:47:34.958381 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958375 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" Sep 08 16:47:34.958517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958384 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" Sep 08 16:47:34.958517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958427 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07620007-2da5-48bf-9f5c-52569cd8437c" containerName="revoke-keys" Sep 08 16:47:34.958517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958435 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07620007-2da5-48bf-9f5c-52569cd8437c" containerName="revoke-keys" Sep 08 16:47:34.958641 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958628 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" Sep 08 16:47:34.958679 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958655 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f252198-3fcd-4324-9cce-42145a51914b" containerName="manager" Sep 08 16:47:34.958679 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958665 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" Sep 08 16:47:34.958679 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958673 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" Sep 08 16:47:34.958792 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.958688 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07620007-2da5-48bf-9f5c-52569cd8437c" containerName="revoke-keys" Sep 08 16:47:34.962727 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.962702 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:34.966923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.966898 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 16:47:34.966923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.966913 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdv66\"" Sep 08 16:47:34.967113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.966934 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 16:47:34.967113 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.966898 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 16:47:34.972889 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:34.972862 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587797bcc9-rn447"] Sep 08 16:47:35.064185 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.064146 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-webhook-cert\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.064399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.064203 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lg4\" (UniqueName: \"kubernetes.io/projected/93328f4b-9bdd-4503-be40-f8e587a1fb64-kube-api-access-c5lg4\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.064399 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.064370 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-metrics-certs\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.165235 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.165203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-webhook-cert\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.165430 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.165265 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lg4\" (UniqueName: \"kubernetes.io/projected/93328f4b-9bdd-4503-be40-f8e587a1fb64-kube-api-access-c5lg4\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.165430 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.165367 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-metrics-certs\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.167816 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.167791 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-webhook-cert\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.167816 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.167810 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-metrics-certs\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.173910 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.173878 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lg4\" (UniqueName: \"kubernetes.io/projected/93328f4b-9bdd-4503-be40-f8e587a1fb64-kube-api-access-c5lg4\") pod \"maas-controller-587797bcc9-rn447\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.274842 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.274747 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:35.551646 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.551620 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587797bcc9-rn447"] Sep 08 16:47:35.553307 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:47:35.553269 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93328f4b_9bdd_4503_be40_f8e587a1fb64.slice/crio-fdc45dfc11e6737f8e01e802bb0a2c80d14dfab13da1c21f96fd3b5bbfd4fc48 WatchSource:0}: Error finding container fdc45dfc11e6737f8e01e802bb0a2c80d14dfab13da1c21f96fd3b5bbfd4fc48: Status 404 returned error can't find the container with id fdc45dfc11e6737f8e01e802bb0a2c80d14dfab13da1c21f96fd3b5bbfd4fc48 Sep 08 16:47:35.554743 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:35.554726 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:47:36.489517 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:36.489483 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-rn447" event={"ID":"93328f4b-9bdd-4503-be40-f8e587a1fb64","Type":"ContainerStarted","Data":"80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69"} Sep 08 16:47:36.489934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:36.489539 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-rn447" event={"ID":"93328f4b-9bdd-4503-be40-f8e587a1fb64","Type":"ContainerStarted","Data":"fdc45dfc11e6737f8e01e802bb0a2c80d14dfab13da1c21f96fd3b5bbfd4fc48"} Sep 08 16:47:36.489934 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:36.489656 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:36.507116 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:36.507053 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-587797bcc9-rn447" podStartSLOduration=2.10294541 podStartE2EDuration="2.50703519s" totalImagesPullingTime="404.08978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:36.505338904 +0000 UTC m=+2090.100337683" watchObservedRunningTime="2026-09-08 16:47:36.50703519 +0000 UTC m=+2090.102033980" Sep 08 16:47:47.496696 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:47.496663 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:54.695895 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:54.695856 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587797bcc9-rn447"] Sep 08 16:47:54.696478 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:54.696057 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-587797bcc9-rn447" podUID="93328f4b-9bdd-4503-be40-f8e587a1fb64" containerName="manager" containerID="cri-o://80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69" gracePeriod=30 Sep 08 16:47:54.946905 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:54.946833 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:55.047923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.047886 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-webhook-cert\") pod \"93328f4b-9bdd-4503-be40-f8e587a1fb64\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " Sep 08 16:47:55.047923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.047930 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-metrics-certs\") pod \"93328f4b-9bdd-4503-be40-f8e587a1fb64\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " Sep 08 16:47:55.048142 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.048013 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5lg4\" (UniqueName: \"kubernetes.io/projected/93328f4b-9bdd-4503-be40-f8e587a1fb64-kube-api-access-c5lg4\") pod \"93328f4b-9bdd-4503-be40-f8e587a1fb64\" (UID: \"93328f4b-9bdd-4503-be40-f8e587a1fb64\") " Sep 08 16:47:55.050023 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.049984 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "93328f4b-9bdd-4503-be40-f8e587a1fb64" (UID: "93328f4b-9bdd-4503-be40-f8e587a1fb64"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:47:55.050146 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.050040 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93328f4b-9bdd-4503-be40-f8e587a1fb64" (UID: "93328f4b-9bdd-4503-be40-f8e587a1fb64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:47:55.050187 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.050161 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93328f4b-9bdd-4503-be40-f8e587a1fb64-kube-api-access-c5lg4" (OuterVolumeSpecName: "kube-api-access-c5lg4") pod "93328f4b-9bdd-4503-be40-f8e587a1fb64" (UID: "93328f4b-9bdd-4503-be40-f8e587a1fb64"). InnerVolumeSpecName "kube-api-access-c5lg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:47:55.149099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.149065 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lg4\" (UniqueName: \"kubernetes.io/projected/93328f4b-9bdd-4503-be40-f8e587a1fb64-kube-api-access-c5lg4\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:47:55.149099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.149095 2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-webhook-cert\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:47:55.149099 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.149105 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93328f4b-9bdd-4503-be40-f8e587a1fb64-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:47:55.600961 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.600923 2555 generic.go:356] "Generic (PLEG): container finished" podID="93328f4b-9bdd-4503-be40-f8e587a1fb64" containerID="80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69" exitCode=0 Sep 08 16:47:55.601186 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.600967 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-rn447" Sep 08 16:47:55.601186 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.601008 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-rn447" event={"ID":"93328f4b-9bdd-4503-be40-f8e587a1fb64","Type":"ContainerDied","Data":"80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69"} Sep 08 16:47:55.601186 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.601060 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-rn447" event={"ID":"93328f4b-9bdd-4503-be40-f8e587a1fb64","Type":"ContainerDied","Data":"fdc45dfc11e6737f8e01e802bb0a2c80d14dfab13da1c21f96fd3b5bbfd4fc48"} Sep 08 16:47:55.601186 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.601085 2555 scope.go:117] "RemoveContainer" containerID="80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69" Sep 08 16:47:55.610990 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.610972 2555 scope.go:117] "RemoveContainer" containerID="80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69" Sep 08 16:47:55.611366 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:47:55.611340 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69\": container with ID starting with 80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69 not found: ID does not exist" containerID="80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69" Sep 08 16:47:55.611437 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.611378 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69"} err="failed to get container status \"80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69\": rpc error: code = NotFound desc = could not find container \"80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69\": container with ID starting with 80102847eb5e9cf7af11cc1411dc2a94403239c69a6080b4209b10577cd83d69 not found: ID does not exist" Sep 08 16:47:55.622785 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.622752 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-587797bcc9-rn447"] Sep 08 16:47:55.626641 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:55.626613 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-587797bcc9-rn447"] Sep 08 16:47:56.411104 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411066 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-587797bcc9-45lc6"] Sep 08 16:47:56.411531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411462 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" Sep 08 16:47:56.411531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411474 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50c25e40-7029-4c25-9e4d-4508b1363ce5" containerName="istio-proxy" Sep 08 16:47:56.411531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411486 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" Sep 08 16:47:56.411531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411492 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" Sep 08 16:47:56.411531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411499 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93328f4b-9bdd-4503-be40-f8e587a1fb64" containerName="manager" Sep 08 16:47:56.411531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411505 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93328f4b-9bdd-4503-be40-f8e587a1fb64" containerName="manager" Sep 08 16:47:56.411723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411576 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5b38cf-abb4-4579-8ca3-21a7685ee0aa" containerName="istio-proxy" Sep 08 16:47:56.411723 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.411586 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93328f4b-9bdd-4503-be40-f8e587a1fb64" containerName="manager" Sep 08 16:47:56.417925 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.417888 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.422285 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.422014 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdv66\"" Sep 08 16:47:56.422285 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.422136 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 16:47:56.422285 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.422214 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 16:47:56.422551 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.422300 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 16:47:56.425909 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.425884 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587797bcc9-45lc6"] Sep 08 16:47:56.460582 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.460537 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a469863c-e51e-4945-95c4-465719b49c81-metrics-certs\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.460775 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.460616 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vph4g\" (UniqueName: \"kubernetes.io/projected/a469863c-e51e-4945-95c4-465719b49c81-kube-api-access-vph4g\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.460775 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.460652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a469863c-e51e-4945-95c4-465719b49c81-webhook-cert\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.561358 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.561317 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vph4g\" (UniqueName: \"kubernetes.io/projected/a469863c-e51e-4945-95c4-465719b49c81-kube-api-access-vph4g\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.561522 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.561364 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a469863c-e51e-4945-95c4-465719b49c81-webhook-cert\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.561522 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.561410 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a469863c-e51e-4945-95c4-465719b49c81-metrics-certs\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.563963 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.563940 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a469863c-e51e-4945-95c4-465719b49c81-metrics-certs\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.564031 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.563962 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a469863c-e51e-4945-95c4-465719b49c81-webhook-cert\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.570331 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.570304 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vph4g\" (UniqueName: \"kubernetes.io/projected/a469863c-e51e-4945-95c4-465719b49c81-kube-api-access-vph4g\") pod \"maas-controller-587797bcc9-45lc6\" (UID: \"a469863c-e51e-4945-95c4-465719b49c81\") " pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.733148 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.733044 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:56.937421 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:56.937384 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93328f4b-9bdd-4503-be40-f8e587a1fb64" path="/var/lib/kubelet/pods/93328f4b-9bdd-4503-be40-f8e587a1fb64/volumes" Sep 08 16:47:57.014378 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:57.014348 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-587797bcc9-45lc6"] Sep 08 16:47:57.015749 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:47:57.015724 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda469863c_e51e_4945_95c4_465719b49c81.slice/crio-b6346e23cf40911114015b21f6f3dca64814856c8b1dcb731c87509a61af02b3 WatchSource:0}: Error finding container b6346e23cf40911114015b21f6f3dca64814856c8b1dcb731c87509a61af02b3: Status 404 returned error can't find the container with id b6346e23cf40911114015b21f6f3dca64814856c8b1dcb731c87509a61af02b3 Sep 08 16:47:57.615872 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:57.615837 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-45lc6" event={"ID":"a469863c-e51e-4945-95c4-465719b49c81","Type":"ContainerStarted","Data":"da75bfbd7a7970fb7feca9d022e4f305a919b3a66d2d797fa6c8602f62f4a2df"} Sep 08 16:47:57.616280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:57.615890 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-587797bcc9-45lc6" event={"ID":"a469863c-e51e-4945-95c4-465719b49c81","Type":"ContainerStarted","Data":"b6346e23cf40911114015b21f6f3dca64814856c8b1dcb731c87509a61af02b3"} Sep 08 16:47:57.616280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:57.615923 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:47:57.634808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:47:57.634750 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-587797bcc9-45lc6" podStartSLOduration=1.165832961 podStartE2EDuration="1.63473587s" totalImagesPullingTime="468.902909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:47:57.631912207 +0000 UTC m=+2111.226910978" watchObservedRunningTime="2026-09-08 16:47:57.63473587 +0000 UTC m=+2111.229734631" Sep 08 16:48:08.625364 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:48:08.625329 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-587797bcc9-45lc6" Sep 08 16:50:47.289505 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:50:47.289471 2555 scope.go:117] "RemoveContainer" containerID="8781bd4c2c1469d1a9c0bc842064e19d3e99bf45b43e99dc110de0568b583455" Sep 08 16:51:42.228932 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:42.228846 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg"] Sep 08 16:51:42.229519 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:42.229123 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" podUID="a405e5a9-1c8a-4d55-a926-5c09a7668c24" containerName="maas-api" containerID="cri-o://7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca" gracePeriod=30 Sep 08 16:51:43.472762 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.472731 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:51:43.576633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.576537 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-maas-api-tls\") pod \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " Sep 08 16:51:43.576633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.576597 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a405e5a9-1c8a-4d55-a926-5c09a7668c24-openshift-service-ca\") pod \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " Sep 08 16:51:43.576633 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.576616 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hgwp\" (UniqueName: \"kubernetes.io/projected/a405e5a9-1c8a-4d55-a926-5c09a7668c24-kube-api-access-6hgwp\") pod \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " Sep 08 16:51:43.576877 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.576653 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-metrics-certs\") pod \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\" (UID: \"a405e5a9-1c8a-4d55-a926-5c09a7668c24\") " Sep 08 16:51:43.577007 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.576975 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a405e5a9-1c8a-4d55-a926-5c09a7668c24-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a405e5a9-1c8a-4d55-a926-5c09a7668c24" (UID: "a405e5a9-1c8a-4d55-a926-5c09a7668c24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:51:43.578834 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.578798 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a405e5a9-1c8a-4d55-a926-5c09a7668c24" (UID: "a405e5a9-1c8a-4d55-a926-5c09a7668c24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:51:43.578956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.578895 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a405e5a9-1c8a-4d55-a926-5c09a7668c24" (UID: "a405e5a9-1c8a-4d55-a926-5c09a7668c24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:51:43.578956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.578935 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a405e5a9-1c8a-4d55-a926-5c09a7668c24-kube-api-access-6hgwp" (OuterVolumeSpecName: "kube-api-access-6hgwp") pod "a405e5a9-1c8a-4d55-a926-5c09a7668c24" (UID: "a405e5a9-1c8a-4d55-a926-5c09a7668c24"). InnerVolumeSpecName "kube-api-access-6hgwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:51:43.678329 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.678280 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-maas-api-tls\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:51:43.678329 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.678322 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a405e5a9-1c8a-4d55-a926-5c09a7668c24-openshift-service-ca\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:51:43.678329 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.678332 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgwp\" (UniqueName: \"kubernetes.io/projected/a405e5a9-1c8a-4d55-a926-5c09a7668c24-kube-api-access-6hgwp\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:51:43.678329 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.678345 2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a405e5a9-1c8a-4d55-a926-5c09a7668c24-metrics-certs\") on node \"ip-10-0-139-234.ec2.internal\" DevicePath \"\"" Sep 08 16:51:43.923335 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.923295 2555 generic.go:356] "Generic (PLEG): container finished" podID="a405e5a9-1c8a-4d55-a926-5c09a7668c24" containerID="7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca" exitCode=0 Sep 08 16:51:43.923509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.923342 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" Sep 08 16:51:43.923509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.923386 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" event={"ID":"a405e5a9-1c8a-4d55-a926-5c09a7668c24","Type":"ContainerDied","Data":"7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca"} Sep 08 16:51:43.923509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.923436 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg" event={"ID":"a405e5a9-1c8a-4d55-a926-5c09a7668c24","Type":"ContainerDied","Data":"b43203250ff1305c1cec3c61649888726a1db779ab101927fcc43ed1f7566ee1"} Sep 08 16:51:43.923509 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.923465 2555 scope.go:117] "RemoveContainer" containerID="7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca" Sep 08 16:51:43.933858 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.933839 2555 scope.go:117] "RemoveContainer" containerID="7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca" Sep 08 16:51:43.934127 ip-10-0-139-234 kubenswrapper[2555]: E0908 16:51:43.934107 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca\": container with ID starting with 7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca not found: ID does not exist" containerID="7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca" Sep 08 16:51:43.934177 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.934135 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca"} err="failed to get container status \"7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca\": rpc error: code = NotFound desc = could not find container \"7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca\": container with ID starting with 7a6c4b3f5d3b3a699c5e4a873a3ce981cc57c7886166eb15212ea22b9f1583ca not found: ID does not exist" Sep 08 16:51:43.946490 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.946455 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg"] Sep 08 16:51:43.950169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:43.950144 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4497bcd-45qkg"] Sep 08 16:51:44.936490 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:44.936455 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a405e5a9-1c8a-4d55-a926-5c09a7668c24" path="/var/lib/kubelet/pods/a405e5a9-1c8a-4d55-a926-5c09a7668c24/volumes" Sep 08 16:51:47.312155 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:47.312121 2555 scope.go:117] "RemoveContainer" containerID="b7419c16754935ce601672cf31f437e363bcc409017ae13490cae12d205ce99d" Sep 08 16:51:47.321041 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:47.321020 2555 scope.go:117] "RemoveContainer" containerID="9b64f4c55d04b26765e1f31092cc3066a45c805bae7a7cf2def1a98e02f6e0a9" Sep 08 16:51:56.027535 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:51:56.027497 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84b96d9cf5-dbzbm_6af6caa1-8971-4149-8823-dfb686d1c507/authorino/0.log" Sep 08 16:52:01.399464 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:01.399427 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587797bcc9-45lc6_a469863c-e51e-4945-95c4-465719b49c81/manager/0.log" Sep 08 16:52:01.507524 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:01.507495 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xzwnj_85d2d37b-1ad6-4901-878c-0d3bd81a9761/server/0.log" Sep 08 16:52:01.627529 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:01.627495 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-m5ndb_3fc8de8d-f903-4533-9b8f-29f992ef2457/manager/0.log" Sep 08 16:52:01.979061 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:01.979034 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-sr52b_1882ea1a-62ba-4afb-a324-92d4c65e2146/copy-manifests/0.log" Sep 08 16:52:01.997417 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:01.997375 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-sr52b_1882ea1a-62ba-4afb-a324-92d4c65e2146/manager/0.log" Sep 08 16:52:02.661041 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:02.661007 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814765-67zhh_ca9534e3-cd3e-47b6-8516-b02373f8b6f0/cleanup/0.log" Sep 08 16:52:03.564048 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:03.564009 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84b96d9cf5-dbzbm_6af6caa1-8971-4149-8823-dfb686d1c507/authorino/0.log" Sep 08 16:52:04.616470 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:04.616433 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl_538b9d79-d7f5-40df-b1b8-3ead45bd262d/istio-proxy/0.log" Sep 08 16:52:04.952398 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:04.952311 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-wpl4m_40dde0df-7dc8-4377-bd6f-7362e4c983e3/kube-auth-proxy/0.log" Sep 08 16:52:05.868229 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:05.868194 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7hq82_ea95f592-cea0-42bd-9f62-42baf20cad23/main/0.log" Sep 08 16:52:06.079160 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:06.079118 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4jsmp_6113ccbe-9a15-4d94-a476-45fe03b49f97/main/0.log" Sep 08 16:52:11.588120 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.588078 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tfrvf/must-gather-blndd"] Sep 08 16:52:11.588599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.588534 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a405e5a9-1c8a-4d55-a926-5c09a7668c24" containerName="maas-api" Sep 08 16:52:11.588599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.588547 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a405e5a9-1c8a-4d55-a926-5c09a7668c24" containerName="maas-api" Sep 08 16:52:11.588689 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.588609 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a405e5a9-1c8a-4d55-a926-5c09a7668c24" containerName="maas-api" Sep 08 16:52:11.591923 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.591903 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:11.594599 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.594573 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfrvf\"/\"openshift-service-ca.crt\"" Sep 08 16:52:11.594720 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.594599 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfrvf\"/\"kube-root-ca.crt\"" Sep 08 16:52:11.601457 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.601426 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/must-gather-blndd"] Sep 08 16:52:11.737280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.737227 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f39d78b-cd02-4735-b044-7490112d4106-must-gather-output\") pod \"must-gather-blndd\" (UID: \"4f39d78b-cd02-4735-b044-7490112d4106\") " pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:11.737471 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.737356 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvk4\" (UniqueName: \"kubernetes.io/projected/4f39d78b-cd02-4735-b044-7490112d4106-kube-api-access-6xvk4\") pod \"must-gather-blndd\" (UID: \"4f39d78b-cd02-4735-b044-7490112d4106\") " pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:11.838313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.838196 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f39d78b-cd02-4735-b044-7490112d4106-must-gather-output\") pod \"must-gather-blndd\" (UID: \"4f39d78b-cd02-4735-b044-7490112d4106\") " pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:11.838313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.838284 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvk4\" (UniqueName: \"kubernetes.io/projected/4f39d78b-cd02-4735-b044-7490112d4106-kube-api-access-6xvk4\") pod \"must-gather-blndd\" (UID: \"4f39d78b-cd02-4735-b044-7490112d4106\") " pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:11.838587 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.838567 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f39d78b-cd02-4735-b044-7490112d4106-must-gather-output\") pod \"must-gather-blndd\" (UID: \"4f39d78b-cd02-4735-b044-7490112d4106\") " pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:11.847345 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.847317 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvk4\" (UniqueName: \"kubernetes.io/projected/4f39d78b-cd02-4735-b044-7490112d4106-kube-api-access-6xvk4\") pod \"must-gather-blndd\" (UID: \"4f39d78b-cd02-4735-b044-7490112d4106\") " pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:11.903174 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:11.903137 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/must-gather-blndd" Sep 08 16:52:12.180388 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:12.180357 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/must-gather-blndd"] Sep 08 16:52:12.182004 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:52:12.181973 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f39d78b_cd02_4735_b044_7490112d4106.slice/crio-84ad48019001499cd8fc1f6ae0092aafe9176cd6f06097e3e36974b0aa042a1c WatchSource:0}: Error finding container 84ad48019001499cd8fc1f6ae0092aafe9176cd6f06097e3e36974b0aa042a1c: Status 404 returned error can't find the container with id 84ad48019001499cd8fc1f6ae0092aafe9176cd6f06097e3e36974b0aa042a1c Sep 08 16:52:13.092616 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:13.092571 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/must-gather-blndd" event={"ID":"4f39d78b-cd02-4735-b044-7490112d4106","Type":"ContainerStarted","Data":"84ad48019001499cd8fc1f6ae0092aafe9176cd6f06097e3e36974b0aa042a1c"} Sep 08 16:52:14.104454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:14.104414 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/must-gather-blndd" event={"ID":"4f39d78b-cd02-4735-b044-7490112d4106","Type":"ContainerStarted","Data":"e943bff440c26bb1d45921e3b4469cee8eec25eeee1be224c824d042f3a66095"} Sep 08 16:52:14.104454 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:14.104459 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/must-gather-blndd" event={"ID":"4f39d78b-cd02-4735-b044-7490112d4106","Type":"ContainerStarted","Data":"1e4db24d910c6c10cdbeb139124cb634e15d992985cd71d732d2980d4d177bed"} Sep 08 16:52:14.121093 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:14.121025 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tfrvf/must-gather-blndd" podStartSLOduration=2.259599359 podStartE2EDuration="3.121000689s" totalImagesPullingTime="861.40133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:14.119883343 +0000 UTC m=+2367.714882105" watchObservedRunningTime="2026-09-08 16:52:14.121000689 +0000 UTC m=+2367.715999450" Sep 08 16:52:14.702045 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:14.702008 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xtsbg_4234fc50-378f-4ac5-a183-0cb15a2365a5/global-pull-secret-syncer/0.log" Sep 08 16:52:14.795157 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:14.795116 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tqbd2_e64c5b01-186f-4002-827e-e2821aad92b0/konnectivity-agent/0.log" Sep 08 16:52:14.900642 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:14.900591 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-234.ec2.internal_bcaf88a8105917bc3decd15ff63fd729/haproxy/0.log" Sep 08 16:52:19.769271 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:19.768228 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84b96d9cf5-dbzbm_6af6caa1-8971-4149-8823-dfb686d1c507/authorino/0.log" Sep 08 16:52:21.507489 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.507453 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_662cc3a7-0499-4eda-82f3-daf4b1cda0f2/alertmanager/0.log" Sep 08 16:52:21.531455 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.531356 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_662cc3a7-0499-4eda-82f3-daf4b1cda0f2/config-reloader/0.log" Sep 08 16:52:21.560332 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.560228 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_662cc3a7-0499-4eda-82f3-daf4b1cda0f2/kube-rbac-proxy-web/0.log" Sep 08 16:52:21.587710 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.587682 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_662cc3a7-0499-4eda-82f3-daf4b1cda0f2/kube-rbac-proxy/0.log" Sep 08 16:52:21.611075 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.611047 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_662cc3a7-0499-4eda-82f3-daf4b1cda0f2/kube-rbac-proxy-metric/0.log" Sep 08 16:52:21.635436 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.635358 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_662cc3a7-0499-4eda-82f3-daf4b1cda0f2/prom-label-proxy/0.log" Sep 08 16:52:21.662688 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.661295 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_662cc3a7-0499-4eda-82f3-daf4b1cda0f2/init-config-reloader/0.log" Sep 08 16:52:21.717076 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:21.717043 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-9vk2j_b979f2af-e275-4d96-a9aa-4c5d72a1410e/cluster-monitoring-operator/0.log" Sep 08 16:52:22.184595 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.184562 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbfjk_fc6e78d5-196b-493b-bd2a-fdecf328afce/node-exporter/0.log" Sep 08 16:52:22.213006 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.212972 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbfjk_fc6e78d5-196b-493b-bd2a-fdecf328afce/kube-rbac-proxy/0.log" Sep 08 16:52:22.236042 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.236006 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pbfjk_fc6e78d5-196b-493b-bd2a-fdecf328afce/init-textfile/0.log" Sep 08 16:52:22.262191 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.262156 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dwplr_1ba98751-9c94-4a95-8c0b-ef0f505acc6e/kube-rbac-proxy-main/0.log" Sep 08 16:52:22.283203 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.283166 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dwplr_1ba98751-9c94-4a95-8c0b-ef0f505acc6e/kube-rbac-proxy-self/0.log" Sep 08 16:52:22.305560 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.305525 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-dwplr_1ba98751-9c94-4a95-8c0b-ef0f505acc6e/openshift-state-metrics/0.log" Sep 08 16:52:22.647064 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.647030 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7488989b8-g42mz_b25f561f-2371-4a04-b73b-4f38cc88fa37/telemeter-client/0.log" Sep 08 16:52:22.708306 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.708232 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7488989b8-g42mz_b25f561f-2371-4a04-b73b-4f38cc88fa37/reload/0.log" Sep 08 16:52:22.731980 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:22.731937 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7488989b8-g42mz_b25f561f-2371-4a04-b73b-4f38cc88fa37/kube-rbac-proxy/0.log" Sep 08 16:52:24.186039 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.186000 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-f9dn5_aae8b909-2a17-49e5-ad81-d0ce8cf3fcfa/networking-console-plugin/0.log" Sep 08 16:52:24.457316 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.457197 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv"] Sep 08 16:52:24.466124 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.466088 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.469147 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.469122 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tfrvf\"/\"default-dockercfg-924bj\"" Sep 08 16:52:24.471474 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.469599 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv"] Sep 08 16:52:24.494907 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.494862 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-lib-modules\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.495164 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.495145 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xw7\" (UniqueName: \"kubernetes.io/projected/a4b7a72c-efdc-45cd-95c0-cf0244064d20-kube-api-access-t6xw7\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.495313 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.495295 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-proc\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.495506 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.495493 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-podres\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.495613 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.495599 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-sys\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.596748 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.596707 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-podres\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.596956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.596758 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-sys\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.596956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.596833 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-lib-modules\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.596956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.596870 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xw7\" (UniqueName: \"kubernetes.io/projected/a4b7a72c-efdc-45cd-95c0-cf0244064d20-kube-api-access-t6xw7\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.596956 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.596911 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-proc\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.597193 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.597080 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-proc\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.597268 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.597204 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-podres\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.597330 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.597267 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-sys\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.597398 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.597358 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b7a72c-efdc-45cd-95c0-cf0244064d20-lib-modules\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.606690 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.606660 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xw7\" (UniqueName: \"kubernetes.io/projected/a4b7a72c-efdc-45cd-95c0-cf0244064d20-kube-api-access-t6xw7\") pod \"perf-node-gather-daemonset-wpzvv\" (UID: \"a4b7a72c-efdc-45cd-95c0-cf0244064d20\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:24.786185 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:24.786083 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:25.092698 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:25.092666 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv"] Sep 08 16:52:25.094515 ip-10-0-139-234 kubenswrapper[2555]: W0908 16:52:25.094467 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4b7a72c_efdc_45cd_95c0_cf0244064d20.slice/crio-c3930275e9026792444dfb93f60cb56ba4c12049589362c5493f424a10ea3f70 WatchSource:0}: Error finding container c3930275e9026792444dfb93f60cb56ba4c12049589362c5493f424a10ea3f70: Status 404 returned error can't find the container with id c3930275e9026792444dfb93f60cb56ba4c12049589362c5493f424a10ea3f70 Sep 08 16:52:25.220459 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:25.220415 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" event={"ID":"a4b7a72c-efdc-45cd-95c0-cf0244064d20","Type":"ContainerStarted","Data":"c3930275e9026792444dfb93f60cb56ba4c12049589362c5493f424a10ea3f70"} Sep 08 16:52:26.061214 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:26.061174 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-qzbg7_5166965d-9ee0-4e8a-b74f-ac3c9735bfcc/volume-data-source-validator/0.log" Sep 08 16:52:26.228280 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:26.228225 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" event={"ID":"a4b7a72c-efdc-45cd-95c0-cf0244064d20","Type":"ContainerStarted","Data":"11671b8f984dd583fe1e90c0a76623714529fb7a5bb272daabb5971aa2e356b2"} Sep 08 16:52:26.228706 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:26.228406 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:26.245941 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:26.245880 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" podStartSLOduration=2.245861822 podStartE2EDuration="2.245861822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:26.244029908 +0000 UTC m=+2379.839028668" watchObservedRunningTime="2026-09-08 16:52:26.245861822 +0000 UTC m=+2379.840860582" Sep 08 16:52:27.199996 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:27.199962 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j4tfc_a5881b84-4529-469e-ac2e-9ff4ca8726ef/dns/0.log" Sep 08 16:52:27.219807 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:27.219776 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j4tfc_a5881b84-4529-469e-ac2e-9ff4ca8726ef/kube-rbac-proxy/0.log" Sep 08 16:52:27.313531 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:27.313488 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxdnw_f0ff108a-6700-47c6-b566-f6e6740810e1/dns-node-resolver/0.log" Sep 08 16:52:27.901971 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:27.901937 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nsbsw_02929572-64d2-492c-8602-3a3dc9bc9701/node-ca/0.log" Sep 08 16:52:28.862483 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:28.862436 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl_538b9d79-d7f5-40df-b1b8-3ead45bd262d/istio-proxy/0.log" Sep 08 16:52:29.339902 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:29.339818 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-wpl4m_40dde0df-7dc8-4377-bd6f-7362e4c983e3/kube-auth-proxy/0.log" Sep 08 16:52:30.360405 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:30.360371 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-765jc_b2e0fedb-4b6f-4a9d-b22c-22d4b7bbb008/serve-healthcheck-canary/0.log" Sep 08 16:52:30.884054 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:30.884018 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-srwhq_a1a5a8ea-d576-42a5-a612-fd894d234b40/insights-operator/0.log" Sep 08 16:52:30.884478 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:30.884446 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-srwhq_a1a5a8ea-d576-42a5-a612-fd894d234b40/insights-operator/1.log" Sep 08 16:52:30.906727 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:30.906687 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d5px9_f4bed502-a9da-4bcb-a714-256171637ea8/kube-rbac-proxy/0.log" Sep 08 16:52:30.928031 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:30.927998 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d5px9_f4bed502-a9da-4bcb-a714-256171637ea8/exporter/0.log" Sep 08 16:52:30.949708 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:30.949661 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d5px9_f4bed502-a9da-4bcb-a714-256171637ea8/extractor/0.log" Sep 08 16:52:32.246546 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:32.246509 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-wpzvv" Sep 08 16:52:33.733895 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:33.733862 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-587797bcc9-45lc6_a469863c-e51e-4945-95c4-465719b49c81/manager/0.log" Sep 08 16:52:33.755169 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:33.755133 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xzwnj_85d2d37b-1ad6-4901-878c-0d3bd81a9761/server/0.log" Sep 08 16:52:33.934145 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:33.934110 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-m5ndb_3fc8de8d-f903-4533-9b8f-29f992ef2457/manager/0.log" Sep 08 16:52:34.201674 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:34.201517 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-sr52b_1882ea1a-62ba-4afb-a324-92d4c65e2146/manager/0.log" Sep 08 16:52:34.221416 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:34.221380 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-sr52b_1882ea1a-62ba-4afb-a324-92d4c65e2146/copy-manifests/0.log" Sep 08 16:52:35.500984 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:35.500937 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78fd8475-jmng8_a2a38ecf-bcd6-4d6d-a370-da4eb7cd7566/manager/0.log" Sep 08 16:52:35.528297 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:35.528262 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kl8nd_b8928908-1aa8-4694-adf0-467fd7388683/openshift-lws-operator/0.log" Sep 08 16:52:36.018007 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:36.017976 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl_538b9d79-d7f5-40df-b1b8-3ead45bd262d/istio-proxy/0.log" Sep 08 16:52:36.465105 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:36.465075 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-wpl4m_40dde0df-7dc8-4377-bd6f-7362e4c983e3/kube-auth-proxy/0.log" Sep 08 16:52:38.034428 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:38.034368 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vpjdl_538b9d79-d7f5-40df-b1b8-3ead45bd262d/istio-proxy/0.log" Sep 08 16:52:38.493705 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:38.493666 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-wpl4m_40dde0df-7dc8-4377-bd6f-7362e4c983e3/kube-auth-proxy/0.log" Sep 08 16:52:40.556235 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:40.556195 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5mj92_0cd2d9b3-c508-4471-b728-d5faf5bcf5b1/migrator/0.log" Sep 08 16:52:40.577288 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:40.577255 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5mj92_0cd2d9b3-c508-4471-b728-d5faf5bcf5b1/graceful-termination/0.log" Sep 08 16:52:40.929112 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:40.929074 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-65p48_1e9c00eb-2239-4e76-9155-8bb822586051/kube-storage-version-migrator-operator/1.log" Sep 08 16:52:40.930827 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:40.930786 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-65p48_1e9c00eb-2239-4e76-9155-8bb822586051/kube-storage-version-migrator-operator/0.log" Sep 08 16:52:42.070165 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.070066 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lj8pc_266f7a5b-0582-4cc3-945f-0229e086fad3/kube-multus-additional-cni-plugins/0.log" Sep 08 16:52:42.091205 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.091174 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lj8pc_266f7a5b-0582-4cc3-945f-0229e086fad3/egress-router-binary-copy/0.log" Sep 08 16:52:42.111378 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.111345 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lj8pc_266f7a5b-0582-4cc3-945f-0229e086fad3/cni-plugins/0.log" Sep 08 16:52:42.132539 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.132508 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lj8pc_266f7a5b-0582-4cc3-945f-0229e086fad3/bond-cni-plugin/0.log" Sep 08 16:52:42.154461 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.154432 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lj8pc_266f7a5b-0582-4cc3-945f-0229e086fad3/routeoverride-cni/0.log" Sep 08 16:52:42.177808 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.177771 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lj8pc_266f7a5b-0582-4cc3-945f-0229e086fad3/whereabouts-cni-bincopy/0.log" Sep 08 16:52:42.200898 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.200866 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lj8pc_266f7a5b-0582-4cc3-945f-0229e086fad3/whereabouts-cni/0.log" Sep 08 16:52:42.423881 ip-10-0-139-234 kubenswrapper[2555]: I0908 16:52:42.423843 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hz52h_a42de246-aa61-4934-96e1-07e4fb87f898/kube-multus/0.log"