Jul 09 14:18:35.330602 ip-10-0-133-55 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 14:18:35.330611 ip-10-0-133-55 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 14:18:35.330618 ip-10-0-133-55 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 14:18:35.330828 ip-10-0-133-55 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 14:18:45.004215 ip-10-0-133-55 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 14:18:45.004230 ip-10-0-133-55 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot aaf223b01feb4cd2b0aa8d204e49cb2a -- Jul 09 14:20:34.790488 ip-10-0-133-55 systemd[1]: Starting Kubernetes Kubelet... Jul 09 14:20:35.321913 ip-10-0-133-55 systemd[1]: Started Kubernetes Kubelet. Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.248629 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253758 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253771 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253776 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253781 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253786 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253790 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253795 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:20:35.336443 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253799 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253803 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253807 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253811 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253815 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253819 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253823 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253827 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253831 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253834 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253840 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253844 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253847 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253851 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253854 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253858 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253862 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253865 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253869 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253872 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:20:35.337562 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253875 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253879 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253883 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253886 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253890 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253895 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253900 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253904 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253908 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253912 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253915 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253920 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253925 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253931 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253934 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253938 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253942 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253945 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253949 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:20:35.338200 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253953 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253956 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253959 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253963 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253966 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253970 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253974 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253977 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253981 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253985 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253988 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253992 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253995 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.253998 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254002 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254006 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254010 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254014 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254018 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254021 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:20:35.338829 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254025 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254029 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254032 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254037 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254041 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254046 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254051 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254055 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254060 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254064 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254068 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254073 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254077 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254081 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254085 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254089 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254094 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254098 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254102 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:20:35.339467 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254106 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254110 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254114 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254117 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254121 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254125 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254129 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254132 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254136 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254140 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254144 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254148 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254151 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.254154 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255103 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255111 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255115 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255119 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255123 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255127 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:20:35.340036 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255131 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255135 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255140 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255144 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255148 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255151 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255155 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255159 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255163 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255167 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255171 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255174 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255178 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255182 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255185 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255189 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255193 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255198 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255202 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255206 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:20:35.340653 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255209 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255212 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255217 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255236 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255240 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255244 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255247 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255250 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255254 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255257 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255261 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255265 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255270 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255274 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255278 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255281 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255285 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255288 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255292 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255296 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:20:35.341483 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255299 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255303 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255308 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255311 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255315 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255318 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255322 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255326 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255329 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255333 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255337 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255342 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255346 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255350 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255353 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255357 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255360 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255364 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255370 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255373 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:20:35.342093 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255377 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255380 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255384 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255387 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255391 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255395 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255399 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255404 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255409 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255413 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255418 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255422 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255425 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255429 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255433 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255437 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255441 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255444 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255448 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:20:35.342706 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255452 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255455 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255458 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255462 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255466 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255469 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255473 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255477 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255481 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255484 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255487 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255491 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255494 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255501 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255958 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255966 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255972 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255976 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:20:35.343480 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255981 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255986 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255990 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255994 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.255998 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256001 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256005 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256009 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256013 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256016 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256022 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256027 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256031 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256034 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256038 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256041 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256045 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256048 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256051 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:20:35.344315 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256055 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256062 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256066 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256069 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256073 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256077 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256081 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256084 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256088 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256092 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256095 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256098 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256103 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256106 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256110 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256114 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256118 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256121 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256125 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256128 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:20:35.345168 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256133 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256136 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256140 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256143 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256147 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256151 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256154 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256158 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256161 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256165 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256168 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256171 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256175 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256181 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256184 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256188 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256192 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256196 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256200 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256203 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:20:35.346076 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256207 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256210 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256214 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256217 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256234 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256238 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256241 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256245 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256249 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256253 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256256 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256259 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256263 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256266 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256270 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256274 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256277 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256281 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256285 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256289 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:20:35.347067 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256292 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256298 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256302 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256306 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256310 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256321 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256325 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256328 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256332 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256335 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256339 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256342 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256346 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256350 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256353 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.256357 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256445 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256460 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256468 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256473 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256482 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 14:20:35.348003 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256487 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256492 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256498 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256502 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256506 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256510 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256515 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256520 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256524 2563 flags.go:64] FLAG: --cgroup-root="" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256528 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256532 2563 flags.go:64] FLAG: --client-ca-file="" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256536 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256540 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256545 2563 flags.go:64] FLAG: --cluster-domain="" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256549 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256553 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256562 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256566 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256571 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256575 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256580 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256584 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256589 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256592 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 14:20:35.349087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256596 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256600 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256604 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256609 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256613 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256618 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256622 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256626 2563 flags.go:64] FLAG: --enable-server="true" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256632 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256638 2563 flags.go:64] FLAG: --event-burst="100" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256642 2563 flags.go:64] FLAG: --event-qps="50" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256646 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256651 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256654 2563 flags.go:64] FLAG: --eviction-hard="" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256663 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256667 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256671 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256675 2563 flags.go:64] FLAG: --eviction-soft="" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256679 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256684 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256688 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256692 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256697 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256701 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256708 2563 flags.go:64] FLAG: --feature-gates="" Jul 09 14:20:35.350193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256713 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256718 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256722 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256726 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256730 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256734 2563 flags.go:64] FLAG: --help="false" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256739 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-55.ec2.internal" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256743 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256748 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256752 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256757 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256761 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256766 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256770 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256774 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256778 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256782 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256787 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256791 2563 flags.go:64] FLAG: --kube-reserved="" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256796 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256800 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256804 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256808 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256812 2563 flags.go:64] FLAG: --lock-file="" Jul 09 14:20:35.351603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256816 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256820 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256824 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256831 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256836 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256840 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256844 2563 flags.go:64] FLAG: --logging-format="text" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256850 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256855 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256859 2563 flags.go:64] FLAG: --manifest-url="" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256863 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256868 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256872 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256878 2563 flags.go:64] FLAG: --max-pods="110" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256882 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256887 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256890 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256894 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256899 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256903 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256908 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256916 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256921 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256925 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 14:20:35.352758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256929 2563 flags.go:64] FLAG: --pod-cidr="" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256934 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256940 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256944 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256948 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256952 2563 flags.go:64] FLAG: --port="10250" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256957 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256961 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d79bf0a1dcfb03a5" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256965 2563 flags.go:64] FLAG: --qos-reserved="" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256969 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256973 2563 flags.go:64] FLAG: --register-node="true" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256978 2563 flags.go:64] FLAG: --register-with-taints="" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256983 2563 flags.go:64] FLAG: --registry-burst="10" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256987 2563 flags.go:64] FLAG: --registry-qps="5" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256991 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.256996 2563 flags.go:64] FLAG: --reserved-memory="" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257003 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257007 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257012 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257016 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257020 2563 flags.go:64] FLAG: --runonce="false" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257024 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257028 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257032 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257036 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 14:20:36.549146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257040 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257044 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257048 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257052 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257056 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257060 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257071 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257076 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257080 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257085 2563 flags.go:64] FLAG: --system-cgroups="" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257089 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257096 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257100 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257104 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257109 2563 flags.go:64] FLAG: --tls-min-version="" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257113 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257117 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257121 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257125 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257129 2563 flags.go:64] FLAG: --v="2" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257134 2563 flags.go:64] FLAG: --version="false" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257139 2563 flags.go:64] FLAG: --vmodule="" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257146 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257153 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257321 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:20:36.568285 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257327 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257331 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257337 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257342 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257346 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257351 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257354 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257357 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257361 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257365 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257368 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257372 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257375 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257379 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257383 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257387 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257390 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257394 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257398 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257402 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:20:36.572991 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257405 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257409 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257412 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257417 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257422 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257426 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257430 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257434 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257437 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257443 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257448 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257452 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257455 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257459 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257463 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257467 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257470 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257474 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257477 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:20:36.573988 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257481 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257484 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257488 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257492 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257495 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257499 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257503 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257506 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257509 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257513 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257516 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257520 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257524 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257527 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257531 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257534 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257538 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257542 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257545 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257548 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257552 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:20:36.574544 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257555 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257560 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257565 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257568 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257572 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257576 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257580 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257584 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257588 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257592 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257595 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257599 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257602 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257606 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257610 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257613 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257617 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257620 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257624 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:20:36.575251 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257629 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257633 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257637 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257641 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257645 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257649 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257653 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257657 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257661 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257664 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257668 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257671 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257675 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257679 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257684 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257689 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257692 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257696 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:20:36.575845 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.257700 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.257708 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.265517 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.265533 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.266256 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.266263 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266323 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266328 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266331 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266334 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266337 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266339 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266342 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266345 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266348 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:20:36.576366 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266351 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266354 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266358 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266362 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266365 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266368 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266371 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266374 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266376 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266379 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266382 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266385 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266388 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266391 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266393 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266396 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266402 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266405 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266408 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:20:36.576813 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266411 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266413 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266416 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266418 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266421 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266424 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266427 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266429 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266432 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266434 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266437 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266439 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266442 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266444 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266447 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266449 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266452 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266454 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266457 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266460 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:20:36.577382 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266462 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266464 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266467 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266469 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266472 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266474 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266478 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266481 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266484 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266488 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266491 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266495 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266497 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266500 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266502 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266505 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266508 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266510 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266513 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266515 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:20:36.577979 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266518 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266520 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266523 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266525 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266527 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266530 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266532 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266535 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266537 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266539 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266542 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266545 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266548 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266550 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266553 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266555 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266557 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266560 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266562 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:20:36.578526 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266565 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266569 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266573 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266576 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266580 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266582 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266585 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266588 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266591 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266594 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266596 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266599 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.266604 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266723 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 14:20:36.579045 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266728 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266731 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266734 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266737 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266740 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266743 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266745 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266748 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266751 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266753 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266757 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266762 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266766 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266769 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266772 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266775 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266778 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266780 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266782 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 14:20:36.579476 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266785 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266787 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266791 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266794 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266797 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266799 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266802 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266804 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266807 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266810 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266812 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266814 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266817 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266820 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266822 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266825 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266828 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266830 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266833 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266835 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 14:20:36.579989 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266838 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266840 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266844 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266847 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266850 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266853 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266855 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266857 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266860 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266862 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266865 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266867 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266870 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266872 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266876 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266879 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266882 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266885 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266887 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 14:20:36.735028 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266889 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266892 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266894 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266897 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266900 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266902 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266905 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266907 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266910 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266912 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266915 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266917 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266919 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266922 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266924 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266927 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266929 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266932 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266934 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266937 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 14:20:36.736003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266940 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266943 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266945 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266948 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266950 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266952 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266955 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266959 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266962 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266965 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266967 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266970 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266972 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266974 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266977 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266981 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266984 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266987 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266990 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 14:20:36.736611 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:35.266993 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.266997 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.267090 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.269824 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.271571 2563 server.go:1019] "Starting client certificate rotation" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.271700 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.271732 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.273849 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.277493 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.285710 2563 log.go:25] "Validated CRI v1 runtime API" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.290777 2563 log.go:25] "Validated CRI v1 image API" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.291583 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 14:20:36.737211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.294180 2563 fs.go:135] Filesystem UUIDs: map[237be131-3b50-4460-b7b8-81ba089d995f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 955fd716-8a75-495f-9ae9-f995be744df1:/dev/nvme0n1p4] Jul 09 14:20:36.737595 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.294199 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.300702 2563 manager.go:216] Machine: {Timestamp:2026-07-09 14:20:35.299357083 +0000 UTC m=+0.380835046 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199676 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a07dc6c4db524bcc69ff522154491 SystemUUID:ec2a07dc-6c4d-b524-bcc6-9ff522154491 BootID:aaf223b0-1feb-4cd2-b0aa-8d204e49cb2a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8d:d4:1c:7b:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8d:d4:1c:7b:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:ab:38:95:f9:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.300794 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.300855 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.304562 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.304590 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.305473 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.305486 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 14:20:36.737680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.305495 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.305512 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.306869 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.307090 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.309987 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310110 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310136 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310144 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310153 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310161 2563 kubelet.go:398] "Adding apiserver pod source" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310169 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310436 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.310451 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.312817 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315081 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315108 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315273 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315285 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315291 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315296 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315304 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.315309 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.316209 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.316217 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.316237 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.316243 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.316251 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.316259 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.318154 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.318162 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.321320 2563 server.go:1268] "Started kubelet" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.321720 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.323368 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.324164 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.324200 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.326343 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.329508 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.329795 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.332334 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.332906 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.332913 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333031 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333085 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333090 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.333622 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 14:20:36.738020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333688 2563 factory.go:55] Registering systemd factory Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333701 2563 factory.go:223] Registration of the systemd container factory successfully Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333857 2563 factory.go:153] Registering CRI-O factory Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333868 2563 factory.go:223] Registration of the crio container factory successfully Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333910 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333929 2563 factory.go:103] Registering Raw factory Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.333941 2563 manager.go:1196] Started watching for new ooms in manager Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.334292 2563 manager.go:349] Starting recovery of all containers Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.335897 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.344846 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.344978 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.345058 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.353571 2563 manager.go:354] Recovery completed Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.358094 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.358844 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.358938 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.360488 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.360514 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.360530 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.358969 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-55.ec2.internal.18c0a4f77e48ccd0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-55.ec2.internal,UID:ip-10-0-133-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-55.ec2.internal,},FirstTimestamp:2026-07-09 14:20:35.321294032 +0000 UTC m=+0.402771974,LastTimestamp:2026-07-09 14:20:35.321294032 +0000 UTC m=+0.402771974,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-55.ec2.internal,}" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.361007 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.361015 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.361023 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.361028 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.361742 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.361775 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.361784 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.362457 2563 policy_none.go:49] "None policy: Start" Jul 09 14:20:36.739418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.362470 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.362480 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.362669 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-55.ec2.internal.18c0a4f7809f17eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-55.ec2.internal,UID:ip-10-0-133-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-55.ec2.internal,},FirstTimestamp:2026-07-09 14:20:35.360503787 +0000 UTC m=+0.441981735,LastTimestamp:2026-07-09 14:20:35.360503787 +0000 UTC m=+0.441981735,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-55.ec2.internal,}" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.363525 2563 policy_none.go:47] "Start" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.370163 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jln8r" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.372543 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-55.ec2.internal.18c0a4f7809f5eef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-55.ec2.internal,UID:ip-10-0-133-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-55.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-55.ec2.internal,},FirstTimestamp:2026-07-09 14:20:35.360521967 +0000 UTC m=+0.441999909,LastTimestamp:2026-07-09 14:20:35.360521967 +0000 UTC m=+0.441999909,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-55.ec2.internal,}" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.377463 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-55.ec2.internal.18c0a4f7809f96bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-55.ec2.internal,UID:ip-10-0-133-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-55.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-55.ec2.internal,},FirstTimestamp:2026-07-09 14:20:35.360536255 +0000 UTC m=+0.442014200,LastTimestamp:2026-07-09 14:20:35.360536255 +0000 UTC m=+0.442014200,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-55.ec2.internal,}" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.380451 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jln8r" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.381943 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-55.ec2.internal.18c0a4f7809f17eb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-55.ec2.internal.18c0a4f7809f17eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-55.ec2.internal,UID:ip-10-0-133-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-55.ec2.internal,},FirstTimestamp:2026-07-09 14:20:35.360503787 +0000 UTC m=+0.441981735,LastTimestamp:2026-07-09 14:20:35.361761786 +0000 UTC m=+0.443239726,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-55.ec2.internal,}" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.384817 2563 manager.go:337] "Starting Device Plugin manager" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.384839 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.384847 2563 server.go:87] "Starting device plugin registration server" Jul 09 14:20:36.740199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.385058 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.385078 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.385147 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.385217 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:35.385239 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.385588 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:35.385632 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.545791 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.546864 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.546890 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.546900 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.546922 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.556817 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.556832 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-55.ec2.internal\": node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.568407 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.568438 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.568453 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.568479 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.572179 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.593986 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.669174 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal"] Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.669222 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.669899 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.669921 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.669930 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671203 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671307 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671332 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671885 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671896 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671906 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671914 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671919 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.671924 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.673196 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.673214 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.673737 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.673758 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.673767 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.694064 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.695142 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f7410766a9ef9617ac405e95724d1bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal\" (UID: \"3f7410766a9ef9617ac405e95724d1bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.695178 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f7410766a9ef9617ac405e95724d1bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal\" (UID: \"3f7410766a9ef9617ac405e95724d1bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.695202 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5340419ec45aff54cbe78dced08dd234-config\") pod \"kube-apiserver-proxy-ip-10-0-133-55.ec2.internal\" (UID: \"5340419ec45aff54cbe78dced08dd234\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.698109 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-55.ec2.internal\" not found" node="ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.704000 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-55.ec2.internal\" not found" node="ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.794447 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.795518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5340419ec45aff54cbe78dced08dd234-config\") pod \"kube-apiserver-proxy-ip-10-0-133-55.ec2.internal\" (UID: \"5340419ec45aff54cbe78dced08dd234\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.795543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f7410766a9ef9617ac405e95724d1bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal\" (UID: \"3f7410766a9ef9617ac405e95724d1bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.795558 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f7410766a9ef9617ac405e95724d1bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal\" (UID: \"3f7410766a9ef9617ac405e95724d1bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.795591 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f7410766a9ef9617ac405e95724d1bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal\" (UID: \"3f7410766a9ef9617ac405e95724d1bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.795636 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f7410766a9ef9617ac405e95724d1bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal\" (UID: \"3f7410766a9ef9617ac405e95724d1bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:36.881978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:36.795633 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5340419ec45aff54cbe78dced08dd234-config\") pod \"kube-apiserver-proxy-ip-10-0-133-55.ec2.internal\" (UID: \"5340419ec45aff54cbe78dced08dd234\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" Jul 09 14:20:37.051613 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.894538 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.051613 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:36.994856 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.051613 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.000072 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:37.051613 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.007664 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" Jul 09 14:20:37.245328 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:37.095363 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.245328 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:37.195805 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.401488 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:37.296193 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.401488 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.331325 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 14:20:37.401488 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.331419 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 14:20:37.401488 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:37.397090 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.517444 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.444590 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 14:15:36 +0000 UTC" deadline="2028-04-22 11:50:06.50219393 +0000 UTC" Jul 09 14:20:37.517444 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.444630 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15669h29m29.057583131s" Jul 09 14:20:37.517444 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:37.497984 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.540247 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.541072 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientMemory" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.541094 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.541109 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeHasSufficientPID" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.573901 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:37.599018 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-55.ec2.internal\" not found" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.692920 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:20:37.706153 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.693902 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" Jul 09 14:20:37.880641 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.711675 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 14:20:37.880641 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.713040 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" Jul 09 14:20:37.880641 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.722246 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 14:20:37.880641 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:37.841543 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:20:38.333555 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.235283 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.372186 2563 apiserver.go:52] "Watching apiserver" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.379904 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.381291 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-9868z","openshift-ovn-kubernetes/ovnkube-node-rzbg4","kube-system/global-pull-secret-syncer-928hb","kube-system/konnectivity-agent-zmmhc","kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal","openshift-multus/multus-additional-cni-plugins-jnrwr","openshift-multus/multus-k49pd","openshift-network-diagnostics/network-check-target-95fkk","openshift-cluster-node-tuning-operator/tuned-95lfw","openshift-dns/node-resolver-sx47j","openshift-image-registry/node-ca-b4jwm","openshift-multus/network-metrics-daemon-fb2rh"] Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.382884 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.382966 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.384097 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.385291 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.385339 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.386406 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.386702 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.386705 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.386822 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-59z6z\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.387750 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.387790 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.388667 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.388876 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.388940 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qvqrj\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.389075 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.390286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.391080 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.391160 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.391721 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85q8v\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.391781 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 14:20:38.429402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.391838 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.392280 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.392630 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sx7xw\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.392750 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.392970 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.393054 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.393151 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.393191 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2q8fs\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.393590 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.393888 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.393591 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.394457 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.395429 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396129 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396149 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396395 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396416 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cncfb\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396416 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396453 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396657 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.396861 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.397708 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wq4mm\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.397918 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.397970 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.398029 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.398505 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.398935 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nxbjn\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.398953 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.399158 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.400833 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.400857 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.400861 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x7dwh\"" Jul 09 14:20:38.432308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.405424 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.415910 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-env-overrides\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.415930 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovn-node-metrics-cert\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.415946 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7474eb-d84f-475a-a445-21a49a350284-cni-binary-copy\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.415960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-cni-bin\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.415978 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysctl-conf\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416000 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4991ea6f-5fcf-498a-8570-0817de45dd00-agent-certs\") pod \"konnectivity-agent-zmmhc\" (UID: \"4991ea6f-5fcf-498a-8570-0817de45dd00\") " pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-systemd-units\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416081 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-netns\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416104 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4qn\" (UniqueName: \"kubernetes.io/projected/df7474eb-d84f-475a-a445-21a49a350284-kube-api-access-cc4qn\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416132 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8dd49727-e4e7-4de1-9765-e0c7f23be495-iptables-alerter-script\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416148 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-systemd\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416168 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-cni-netd\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.433368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416198 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e527178e-badd-4625-a67a-f5e2904fbc82-kubelet-config\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-kubernetes\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416224 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abec23fe-04cb-43a0-917e-721e2f4e4ce0-tmp\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416240 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416273 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416311 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-registration-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-device-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416375 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-system-cni-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysctl-d\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416435 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-log-socket\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416459 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-socket-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416480 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-host\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416502 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-run-netns\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416525 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-var-lib-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.562202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-socket-dir-parent\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416570 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-systemd\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416599 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysconfig\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416656 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4991ea6f-5fcf-498a-8570-0817de45dd00-konnectivity-ca\") pod \"konnectivity-agent-zmmhc\" (UID: \"4991ea6f-5fcf-498a-8570-0817de45dd00\") " pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416681 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-slash\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416696 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-run-ovn-kubernetes\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416728 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48j8b\" (UniqueName: \"kubernetes.io/projected/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-kube-api-access-48j8b\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-run\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416786 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-var-lib-kubelet\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-cni-multus\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416838 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-sys\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416858 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-tuned\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovnkube-script-lib\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-k8s-cni-cncf-io\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.563242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416926 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-etc-kubernetes\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4w8c\" (UniqueName: \"kubernetes.io/projected/8dd49727-e4e7-4de1-9765-e0c7f23be495-kube-api-access-w4w8c\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416962 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ca81f21-58d3-478f-a38e-afd97d8aff04-host\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-cni-bin\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.416989 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417004 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6sk\" (UniqueName: \"kubernetes.io/projected/7c264538-ef22-42cf-bf5f-ec59ef08f27a-kube-api-access-2l6sk\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-os-release\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-hostroot\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417050 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnswn\" (UniqueName: \"kubernetes.io/projected/7ca81f21-58d3-478f-a38e-afd97d8aff04-kube-api-access-dnswn\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417067 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-sys-fs\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417118 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7474eb-d84f-475a-a445-21a49a350284-multus-daemon-config\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.563721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhlp\" (UniqueName: \"kubernetes.io/projected/abec23fe-04cb-43a0-917e-721e2f4e4ce0-kube-api-access-2qhlp\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417249 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmx72\" (UniqueName: \"kubernetes.io/projected/4154366d-8c15-403c-8df9-cceea3424319-kube-api-access-cmx72\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417280 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-kubelet\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e527178e-badd-4625-a67a-f5e2904fbc82-dbus\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-cnibin\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417347 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-conf-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dd49727-e4e7-4de1-9765-e0c7f23be495-host-slash\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-lib-modules\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417411 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-node-log\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417433 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-cni-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ca81f21-58d3-478f-a38e-afd97d8aff04-serviceca\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-etc-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417542 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-multus-certs\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.564238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417778 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-modprobe-d\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417799 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-kubelet\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417817 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-ovn\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.417836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovnkube-config\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.426353 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2795c" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.434936 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2795c" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.451778 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7410766a9ef9617ac405e95724d1bd.slice/crio-9bad5d95cb4d52e2b6feb509e77f944f731ab9aa773df0c6e36f1f5c4a931f8c WatchSource:0}: Error finding container 9bad5d95cb4d52e2b6feb509e77f944f731ab9aa773df0c6e36f1f5c4a931f8c: Status 404 returned error can't find the container with id 9bad5d95cb4d52e2b6feb509e77f944f731ab9aa773df0c6e36f1f5c4a931f8c Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.451980 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5340419ec45aff54cbe78dced08dd234.slice/crio-05e5e9690378c8d854cba0beea11e9716c4711ae19b7188e198ea98359f15ec4 WatchSource:0}: Error finding container 05e5e9690378c8d854cba0beea11e9716c4711ae19b7188e198ea98359f15ec4: Status 404 returned error can't find the container with id 05e5e9690378c8d854cba0beea11e9716c4711ae19b7188e198ea98359f15ec4 Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.456784 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.494481 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519491 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-slash\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-slash\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-run-ovn-kubernetes\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519638 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48j8b\" (UniqueName: \"kubernetes.io/projected/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-kube-api-access-48j8b\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-system-cni-dir\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-run\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.564720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519685 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-var-lib-kubelet\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519697 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-run-ovn-kubernetes\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519700 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-cni-multus\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519758 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-var-lib-kubelet\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519766 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-run\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-sys\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519809 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-cni-multus\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519841 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-tuned\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovnkube-script-lib\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-k8s-cni-cncf-io\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519904 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-etc-kubernetes\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519953 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-etc-kubernetes\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.519998 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-sys\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520029 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4w8c\" (UniqueName: \"kubernetes.io/projected/8dd49727-e4e7-4de1-9765-e0c7f23be495-kube-api-access-w4w8c\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ca81f21-58d3-478f-a38e-afd97d8aff04-host\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.565299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520090 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ca81f21-58d3-478f-a38e-afd97d8aff04-host\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520088 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-k8s-cni-cncf-io\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520103 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-cni-bin\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520158 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6sk\" (UniqueName: \"kubernetes.io/projected/7c264538-ef22-42cf-bf5f-ec59ef08f27a-kube-api-access-2l6sk\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-os-release\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520172 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-cni-bin\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-hostroot\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520221 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfxm\" (UniqueName: \"kubernetes.io/projected/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-kube-api-access-djfxm\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.520241 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520246 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnswn\" (UniqueName: \"kubernetes.io/projected/7ca81f21-58d3-478f-a38e-afd97d8aff04-kube-api-access-dnswn\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520247 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-os-release\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.565838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520273 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-hostroot\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.520303 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret podName:e527178e-badd-4625-a67a-f5e2904fbc82 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:39.020284744 +0000 UTC m=+3.041367999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret") pod "global-pull-secret-syncer-928hb" (UID: "e527178e-badd-4625-a67a-f5e2904fbc82") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-sys-fs\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7474eb-d84f-475a-a445-21a49a350284-multus-daemon-config\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520324 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520411 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520418 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-sys-fs\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520479 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovnkube-script-lib\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520479 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhlp\" (UniqueName: \"kubernetes.io/projected/abec23fe-04cb-43a0-917e-721e2f4e4ce0-kube-api-access-2qhlp\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520547 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.566306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmx72\" (UniqueName: \"kubernetes.io/projected/4154366d-8c15-403c-8df9-cceea3424319-kube-api-access-cmx72\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520640 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-kubelet\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520670 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e527178e-badd-4625-a67a-f5e2904fbc82-dbus\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-cnibin\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-kubelet\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-conf-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dd49727-e4e7-4de1-9765-e0c7f23be495-host-slash\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520772 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-conf-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520781 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-lib-modules\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520783 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-cnibin\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520786 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dd49727-e4e7-4de1-9765-e0c7f23be495-host-slash\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-node-log\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520813 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e527178e-badd-4625-a67a-f5e2904fbc82-dbus\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-cni-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.566887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520839 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-node-log\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520853 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5d7\" (UniqueName: \"kubernetes.io/projected/d6a5d9db-5636-48cd-bb96-76b36e182c4b-kube-api-access-zg5d7\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520866 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7474eb-d84f-475a-a445-21a49a350284-multus-daemon-config\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520872 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-lib-modules\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520896 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-cni-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ca81f21-58d3-478f-a38e-afd97d8aff04-serviceca\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-etc-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-multus-certs\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520979 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-modprobe-d\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.520979 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-etc-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521000 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-multus-certs\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521031 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-kubelet\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-ovn\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-modprobe-d\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.567311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovnkube-config\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521074 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-kubelet\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-env-overrides\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521100 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-ovn\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovn-node-metrics-cert\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7474eb-d84f-475a-a445-21a49a350284-cni-binary-copy\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-cni-bin\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521158 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ca81f21-58d3-478f-a38e-afd97d8aff04-serviceca\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysctl-conf\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4991ea6f-5fcf-498a-8570-0817de45dd00-agent-certs\") pod \"konnectivity-agent-zmmhc\" (UID: \"4991ea6f-5fcf-498a-8570-0817de45dd00\") " pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521203 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-var-lib-cni-bin\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-systemd-units\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521233 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-netns\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521280 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4qn\" (UniqueName: \"kubernetes.io/projected/df7474eb-d84f-475a-a445-21a49a350284-kube-api-access-cc4qn\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.567754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521296 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysctl-conf\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8dd49727-e4e7-4de1-9765-e0c7f23be495-iptables-alerter-script\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521343 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cnibin\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-systemd\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-cni-netd\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521437 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e527178e-badd-4625-a67a-f5e2904fbc82-kubelet-config\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521467 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a5d9db-5636-48cd-bb96-76b36e182c4b-tmp-dir\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521491 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521495 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-env-overrides\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-kubernetes\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521547 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovnkube-config\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521562 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-systemd\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521572 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-kubernetes\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abec23fe-04cb-43a0-917e-721e2f4e4ce0-tmp\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521601 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7474eb-d84f-475a-a445-21a49a350284-cni-binary-copy\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-host-run-netns\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521664 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-systemd-units\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.521688 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521725 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-registration-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.521746 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs podName:4154366d-8c15-403c-8df9-cceea3424319 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:39.021727858 +0000 UTC m=+3.042811112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs") pod "network-metrics-daemon-fb2rh" (UID: "4154366d-8c15-403c-8df9-cceea3424319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521750 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e527178e-badd-4625-a67a-f5e2904fbc82-kubelet-config\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521688 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-registration-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521771 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-device-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-cni-netd\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521863 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-system-cni-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.568495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-device-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6a5d9db-5636-48cd-bb96-76b36e182c4b-hosts-file\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521930 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-system-cni-dir\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.521965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysctl-d\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522007 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-log-socket\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-socket-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522066 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysctl-d\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522084 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-host\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522111 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-log-socket\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522146 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-run-netns\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522167 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c264538-ef22-42cf-bf5f-ec59ef08f27a-socket-dir\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-var-lib-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522205 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-host-run-netns\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522214 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-host\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-var-lib-openvswitch\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.568930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522227 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8dd49727-e4e7-4de1-9765-e0c7f23be495-iptables-alerter-script\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-socket-dir-parent\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522267 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-os-release\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-systemd\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522314 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysconfig\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522335 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7474eb-d84f-475a-a445-21a49a350284-multus-socket-dir-parent\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522338 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4991ea6f-5fcf-498a-8570-0817de45dd00-konnectivity-ca\") pod \"konnectivity-agent-zmmhc\" (UID: \"4991ea6f-5fcf-498a-8570-0817de45dd00\") " pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522371 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-run-systemd\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.522424 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-sysconfig\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.523422 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abec23fe-04cb-43a0-917e-721e2f4e4ce0-etc-tuned\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.523441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4991ea6f-5fcf-498a-8570-0817de45dd00-konnectivity-ca\") pod \"konnectivity-agent-zmmhc\" (UID: \"4991ea6f-5fcf-498a-8570-0817de45dd00\") " pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.523589 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-ovn-node-metrics-cert\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.523828 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4991ea6f-5fcf-498a-8570-0817de45dd00-agent-certs\") pod \"konnectivity-agent-zmmhc\" (UID: \"4991ea6f-5fcf-498a-8570-0817de45dd00\") " pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.524027 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abec23fe-04cb-43a0-917e-721e2f4e4ce0-tmp\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.528776 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48j8b\" (UniqueName: \"kubernetes.io/projected/b3aca0a7-2162-4481-bd9d-f2dee1a6fa33-kube-api-access-48j8b\") pod \"ovnkube-node-rzbg4\" (UID: \"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33\") " pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.569373 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.530652 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.530665 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.530673 2563 projected.go:196] Error preparing data for projected volume kube-api-access-6vdnw for pod openshift-network-diagnostics/network-check-target-95fkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:38.530722 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw podName:7a682c4e-eed0-4463-b5d3-b91351ebbe96 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:39.030711064 +0000 UTC m=+3.051794311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6vdnw" (UniqueName: "kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw") pod "network-check-target-95fkk" (UID: "7a682c4e-eed0-4463-b5d3-b91351ebbe96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.531005 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4w8c\" (UniqueName: \"kubernetes.io/projected/8dd49727-e4e7-4de1-9765-e0c7f23be495-kube-api-access-w4w8c\") pod \"iptables-alerter-9868z\" (UID: \"8dd49727-e4e7-4de1-9765-e0c7f23be495\") " pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.531440 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhlp\" (UniqueName: \"kubernetes.io/projected/abec23fe-04cb-43a0-917e-721e2f4e4ce0-kube-api-access-2qhlp\") pod \"tuned-95lfw\" (UID: \"abec23fe-04cb-43a0-917e-721e2f4e4ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.531594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmx72\" (UniqueName: \"kubernetes.io/projected/4154366d-8c15-403c-8df9-cceea3424319-kube-api-access-cmx72\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.532891 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnswn\" (UniqueName: \"kubernetes.io/projected/7ca81f21-58d3-478f-a38e-afd97d8aff04-kube-api-access-dnswn\") pod \"node-ca-b4jwm\" (UID: \"7ca81f21-58d3-478f-a38e-afd97d8aff04\") " pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.533250 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6sk\" (UniqueName: \"kubernetes.io/projected/7c264538-ef22-42cf-bf5f-ec59ef08f27a-kube-api-access-2l6sk\") pod \"aws-ebs-csi-driver-node-ffp5k\" (UID: \"7c264538-ef22-42cf-bf5f-ec59ef08f27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.569840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.533303 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4qn\" (UniqueName: \"kubernetes.io/projected/df7474eb-d84f-475a-a445-21a49a350284-kube-api-access-cc4qn\") pod \"multus-k49pd\" (UID: \"df7474eb-d84f-475a-a445-21a49a350284\") " pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.571966 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.571925 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" event={"ID":"3f7410766a9ef9617ac405e95724d1bd","Type":"ContainerStarted","Data":"9bad5d95cb4d52e2b6feb509e77f944f731ab9aa773df0c6e36f1f5c4a931f8c"} Jul 09 14:20:38.572857 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.572840 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" event={"ID":"5340419ec45aff54cbe78dced08dd234","Type":"ContainerStarted","Data":"05e5e9690378c8d854cba0beea11e9716c4711ae19b7188e198ea98359f15ec4"} Jul 09 14:20:38.623061 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6a5d9db-5636-48cd-bb96-76b36e182c4b-hosts-file\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.623061 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623053 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-os-release\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623072 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-system-cni-dir\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623096 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfxm\" (UniqueName: \"kubernetes.io/projected/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-kube-api-access-djfxm\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623139 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6a5d9db-5636-48cd-bb96-76b36e182c4b-hosts-file\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-os-release\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623161 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-system-cni-dir\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5d7\" (UniqueName: \"kubernetes.io/projected/d6a5d9db-5636-48cd-bb96-76b36e182c4b-kube-api-access-zg5d7\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.623473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623206 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623247 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cnibin\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623285 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cnibin\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623333 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a5d9db-5636-48cd-bb96-76b36e182c4b-tmp-dir\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.623473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623672 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623568 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623704 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623669 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a5d9db-5636-48cd-bb96-76b36e182c4b-tmp-dir\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.623704 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623779 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623761 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.623807 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.623761 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.636419 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.636392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfxm\" (UniqueName: \"kubernetes.io/projected/c5c393e5-2ad0-433a-9373-ff8aea28ed0c-kube-api-access-djfxm\") pod \"multus-additional-cni-plugins-jnrwr\" (UID: \"c5c393e5-2ad0-433a-9373-ff8aea28ed0c\") " pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.638476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.638459 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5d7\" (UniqueName: \"kubernetes.io/projected/d6a5d9db-5636-48cd-bb96-76b36e182c4b-kube-api-access-zg5d7\") pod \"node-resolver-sx47j\" (UID: \"d6a5d9db-5636-48cd-bb96-76b36e182c4b\") " pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.719213 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.719162 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-95lfw" Jul 09 14:20:38.723089 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.723075 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:38.728402 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.728385 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabec23fe_04cb_43a0_917e_721e2f4e4ce0.slice/crio-bfda6227635923d4884203c20f8f144de2673167b76e3ca594a4b933f0628a59 WatchSource:0}: Error finding container bfda6227635923d4884203c20f8f144de2673167b76e3ca594a4b933f0628a59: Status 404 returned error can't find the container with id bfda6227635923d4884203c20f8f144de2673167b76e3ca594a4b933f0628a59 Jul 09 14:20:38.731207 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.731190 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4991ea6f_5fcf_498a_8570_0817de45dd00.slice/crio-4f3e6fb43ffe4a536806da3db1b259b62d118b0b1c59d5b032db4f6d09891027 WatchSource:0}: Error finding container 4f3e6fb43ffe4a536806da3db1b259b62d118b0b1c59d5b032db4f6d09891027: Status 404 returned error can't find the container with id 4f3e6fb43ffe4a536806da3db1b259b62d118b0b1c59d5b032db4f6d09891027 Jul 09 14:20:38.765674 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.765651 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" Jul 09 14:20:38.768784 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.768770 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k49pd" Jul 09 14:20:38.773678 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.773659 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c264538_ef22_42cf_bf5f_ec59ef08f27a.slice/crio-65baa800d4ccee49695c01371bb026b5cb65c6cf511225d6ffa9b9717eba6b79 WatchSource:0}: Error finding container 65baa800d4ccee49695c01371bb026b5cb65c6cf511225d6ffa9b9717eba6b79: Status 404 returned error can't find the container with id 65baa800d4ccee49695c01371bb026b5cb65c6cf511225d6ffa9b9717eba6b79 Jul 09 14:20:38.774209 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.774194 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b4jwm" Jul 09 14:20:38.776964 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.776944 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7474eb_d84f_475a_a445_21a49a350284.slice/crio-792ed87c513918e4e72d94b0b6566de9d93fc74480a38429d4b1add5b18f9ce3 WatchSource:0}: Error finding container 792ed87c513918e4e72d94b0b6566de9d93fc74480a38429d4b1add5b18f9ce3: Status 404 returned error can't find the container with id 792ed87c513918e4e72d94b0b6566de9d93fc74480a38429d4b1add5b18f9ce3 Jul 09 14:20:38.780120 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.780103 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:20:38.781320 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.781302 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca81f21_58d3_478f_a38e_afd97d8aff04.slice/crio-95cf4e54e58653792d43280ba41ad5dfe7af1e2eeeacbfdf614952e7218dc0fa WatchSource:0}: Error finding container 95cf4e54e58653792d43280ba41ad5dfe7af1e2eeeacbfdf614952e7218dc0fa: Status 404 returned error can't find the container with id 95cf4e54e58653792d43280ba41ad5dfe7af1e2eeeacbfdf614952e7218dc0fa Jul 09 14:20:38.788005 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.787989 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aca0a7_2162_4481_bd9d_f2dee1a6fa33.slice/crio-58eaab39097b45402451524422a182d1aad35f428e13935200c590d3d2a863c7 WatchSource:0}: Error finding container 58eaab39097b45402451524422a182d1aad35f428e13935200c590d3d2a863c7: Status 404 returned error can't find the container with id 58eaab39097b45402451524422a182d1aad35f428e13935200c590d3d2a863c7 Jul 09 14:20:38.792033 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.792017 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9868z" Jul 09 14:20:38.795738 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.795721 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" Jul 09 14:20:38.798410 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.798394 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd49727_e4e7_4de1_9765_e0c7f23be495.slice/crio-d115cec3d14c2148918a2ceae54f1d7720dd524819f8aeaf2d2b13d746e8f465 WatchSource:0}: Error finding container d115cec3d14c2148918a2ceae54f1d7720dd524819f8aeaf2d2b13d746e8f465: Status 404 returned error can't find the container with id d115cec3d14c2148918a2ceae54f1d7720dd524819f8aeaf2d2b13d746e8f465 Jul 09 14:20:38.799434 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:38.799422 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sx47j" Jul 09 14:20:38.803019 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.803002 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c393e5_2ad0_433a_9373_ff8aea28ed0c.slice/crio-f44603bb07dfdb90af8644aaa2ad212ce4caf85e630325ae846af372725073b6 WatchSource:0}: Error finding container f44603bb07dfdb90af8644aaa2ad212ce4caf85e630325ae846af372725073b6: Status 404 returned error can't find the container with id f44603bb07dfdb90af8644aaa2ad212ce4caf85e630325ae846af372725073b6 Jul 09 14:20:38.806836 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:20:38.806819 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a5d9db_5636_48cd_bb96_76b36e182c4b.slice/crio-b16e1b68aa0441cd89d97340a6f3581b6ce94cadae9ae5c4ed4ba1893f48bceb WatchSource:0}: Error finding container b16e1b68aa0441cd89d97340a6f3581b6ce94cadae9ae5c4ed4ba1893f48bceb: Status 404 returned error can't find the container with id b16e1b68aa0441cd89d97340a6f3581b6ce94cadae9ae5c4ed4ba1893f48bceb Jul 09 14:20:39.026314 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.026263 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:39.026407 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.026310 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:39.026407 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.026397 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:39.026407 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.026403 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:39.026528 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.026441 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs podName:4154366d-8c15-403c-8df9-cceea3424319 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:40.026428458 +0000 UTC m=+4.047511701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs") pod "network-metrics-daemon-fb2rh" (UID: "4154366d-8c15-403c-8df9-cceea3424319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:39.026528 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.026453 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret podName:e527178e-badd-4625-a67a-f5e2904fbc82 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:40.026447812 +0000 UTC m=+4.047531054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret") pod "global-pull-secret-syncer-928hb" (UID: "e527178e-badd-4625-a67a-f5e2904fbc82") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:39.127240 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.127221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:39.127360 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.127345 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:20:39.127425 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.127368 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:20:39.127425 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.127378 2563 projected.go:196] Error preparing data for projected volume kube-api-access-6vdnw for pod openshift-network-diagnostics/network-check-target-95fkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:39.127540 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.127430 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw podName:7a682c4e-eed0-4463-b5d3-b91351ebbe96 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:40.127412865 +0000 UTC m=+4.148496109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vdnw" (UniqueName: "kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw") pod "network-check-target-95fkk" (UID: "7a682c4e-eed0-4463-b5d3-b91351ebbe96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:39.436538 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.436437 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:15:38 +0000 UTC" deadline="2028-01-08 06:15:17.308459129 +0000 UTC" Jul 09 14:20:39.436538 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.436523 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13143h54m37.871940772s" Jul 09 14:20:39.569143 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.569115 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:39.569306 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.569243 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:39.569677 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.569656 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:39.569767 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:39.569749 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:39.581342 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.581312 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sx47j" event={"ID":"d6a5d9db-5636-48cd-bb96-76b36e182c4b","Type":"ContainerStarted","Data":"b16e1b68aa0441cd89d97340a6f3581b6ce94cadae9ae5c4ed4ba1893f48bceb"} Jul 09 14:20:39.584059 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.584035 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerStarted","Data":"f44603bb07dfdb90af8644aaa2ad212ce4caf85e630325ae846af372725073b6"} Jul 09 14:20:39.593654 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.593627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k49pd" event={"ID":"df7474eb-d84f-475a-a445-21a49a350284","Type":"ContainerStarted","Data":"792ed87c513918e4e72d94b0b6566de9d93fc74480a38429d4b1add5b18f9ce3"} Jul 09 14:20:39.603841 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.603817 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" event={"ID":"7c264538-ef22-42cf-bf5f-ec59ef08f27a","Type":"ContainerStarted","Data":"65baa800d4ccee49695c01371bb026b5cb65c6cf511225d6ffa9b9717eba6b79"} Jul 09 14:20:39.606206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.606182 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-95lfw" event={"ID":"abec23fe-04cb-43a0-917e-721e2f4e4ce0","Type":"ContainerStarted","Data":"bfda6227635923d4884203c20f8f144de2673167b76e3ca594a4b933f0628a59"} Jul 09 14:20:39.614857 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.614784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9868z" event={"ID":"8dd49727-e4e7-4de1-9765-e0c7f23be495","Type":"ContainerStarted","Data":"d115cec3d14c2148918a2ceae54f1d7720dd524819f8aeaf2d2b13d746e8f465"} Jul 09 14:20:39.619924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.619901 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"58eaab39097b45402451524422a182d1aad35f428e13935200c590d3d2a863c7"} Jul 09 14:20:39.623093 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.623070 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b4jwm" event={"ID":"7ca81f21-58d3-478f-a38e-afd97d8aff04","Type":"ContainerStarted","Data":"95cf4e54e58653792d43280ba41ad5dfe7af1e2eeeacbfdf614952e7218dc0fa"} Jul 09 14:20:39.630643 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:39.630606 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmmhc" event={"ID":"4991ea6f-5fcf-498a-8570-0817de45dd00","Type":"ContainerStarted","Data":"4f3e6fb43ffe4a536806da3db1b259b62d118b0b1c59d5b032db4f6d09891027"} Jul 09 14:20:40.033022 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:40.032987 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:40.033240 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:40.033049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:40.033240 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.033190 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:40.033357 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.033251 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret podName:e527178e-badd-4625-a67a-f5e2904fbc82 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:42.033231563 +0000 UTC m=+6.054314828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret") pod "global-pull-secret-syncer-928hb" (UID: "e527178e-badd-4625-a67a-f5e2904fbc82") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:40.033661 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.033643 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:40.033749 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.033698 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs podName:4154366d-8c15-403c-8df9-cceea3424319 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:42.033681115 +0000 UTC m=+6.054764368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs") pod "network-metrics-daemon-fb2rh" (UID: "4154366d-8c15-403c-8df9-cceea3424319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:40.133655 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:40.133361 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:40.133655 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.133540 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:20:40.133655 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.133561 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:20:40.133655 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.133573 2563 projected.go:196] Error preparing data for projected volume kube-api-access-6vdnw for pod openshift-network-diagnostics/network-check-target-95fkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:40.133655 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.133666 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw podName:7a682c4e-eed0-4463-b5d3-b91351ebbe96 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:42.133612707 +0000 UTC m=+6.154695963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vdnw" (UniqueName: "kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw") pod "network-check-target-95fkk" (UID: "7a682c4e-eed0-4463-b5d3-b91351ebbe96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:40.438424 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:40.437365 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 14:15:38 +0000 UTC" deadline="2028-03-16 19:36:02.75774987 +0000 UTC" Jul 09 14:20:40.438424 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:40.437403 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14789h15m22.320351721s" Jul 09 14:20:40.577767 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:40.577736 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:40.577947 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:40.577912 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:41.568867 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:41.568794 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:41.569317 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:41.568917 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:41.569317 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:41.569284 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:41.569422 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:41.569379 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:42.048421 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:42.048348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:42.048421 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:42.048406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:42.048687 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.048546 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:42.048687 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.048611 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret podName:e527178e-badd-4625-a67a-f5e2904fbc82 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:46.048593731 +0000 UTC m=+10.069676979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret") pod "global-pull-secret-syncer-928hb" (UID: "e527178e-badd-4625-a67a-f5e2904fbc82") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:42.048687 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.048547 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:42.048853 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.048694 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs podName:4154366d-8c15-403c-8df9-cceea3424319 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:46.048676041 +0000 UTC m=+10.069759287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs") pod "network-metrics-daemon-fb2rh" (UID: "4154366d-8c15-403c-8df9-cceea3424319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:42.148713 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:42.148681 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:42.148891 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.148840 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:20:42.148891 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.148860 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:20:42.148891 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.148871 2563 projected.go:196] Error preparing data for projected volume kube-api-access-6vdnw for pod openshift-network-diagnostics/network-check-target-95fkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:42.149043 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.148917 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw podName:7a682c4e-eed0-4463-b5d3-b91351ebbe96 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:46.148901477 +0000 UTC m=+10.169984737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vdnw" (UniqueName: "kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw") pod "network-check-target-95fkk" (UID: "7a682c4e-eed0-4463-b5d3-b91351ebbe96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:42.569998 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:42.569478 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:42.569998 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:42.569642 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:43.569504 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:43.569474 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:43.569691 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:43.569474 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:43.569691 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:43.569628 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:43.569691 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:43.569669 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:44.569751 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:44.569649 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:44.570196 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:44.569800 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:45.568732 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:45.568654 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:45.568904 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:45.568874 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:45.568961 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:45.568929 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:45.569069 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:45.569044 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:46.076053 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:46.076003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:46.076492 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:46.076068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:46.076492 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.076148 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:46.076492 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.076179 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:46.076492 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.076206 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs podName:4154366d-8c15-403c-8df9-cceea3424319 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:54.07618831 +0000 UTC m=+18.097271567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs") pod "network-metrics-daemon-fb2rh" (UID: "4154366d-8c15-403c-8df9-cceea3424319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:46.076492 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.076224 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret podName:e527178e-badd-4625-a67a-f5e2904fbc82 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:54.076214553 +0000 UTC m=+18.097297796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret") pod "global-pull-secret-syncer-928hb" (UID: "e527178e-badd-4625-a67a-f5e2904fbc82") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:46.177467 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:46.177421 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:46.177654 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.177631 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:20:46.177749 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.177662 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:20:46.177749 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.177676 2563 projected.go:196] Error preparing data for projected volume kube-api-access-6vdnw for pod openshift-network-diagnostics/network-check-target-95fkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:46.177749 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.177741 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw podName:7a682c4e-eed0-4463-b5d3-b91351ebbe96 nodeName:}" failed. No retries permitted until 2026-07-09 14:20:54.177723892 +0000 UTC m=+18.198807141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vdnw" (UniqueName: "kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw") pod "network-check-target-95fkk" (UID: "7a682c4e-eed0-4463-b5d3-b91351ebbe96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:46.569739 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:46.569662 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:46.569893 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:46.569757 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:47.569671 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:47.569526 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:47.570111 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:47.569696 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:47.570111 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:47.569953 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:47.570111 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:47.570068 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:48.570975 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:48.570940 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:48.571409 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:48.571173 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:49.569848 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.569564 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:49.569955 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:49.569910 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:49.570200 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.569579 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:49.570200 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:49.570167 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:49.667846 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.667788 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b4jwm" event={"ID":"7ca81f21-58d3-478f-a38e-afd97d8aff04","Type":"ContainerStarted","Data":"e5877518769cf2dd6d3b81ef7c140adf77c26766b325959fd1d7c644794d83a1"} Jul 09 14:20:49.669590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.669564 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmmhc" event={"ID":"4991ea6f-5fcf-498a-8570-0817de45dd00","Type":"ContainerStarted","Data":"908ebddd6c6050fc3e299a38aac1120b43fdd657c4c0c800e5f7ebcd9256b35b"} Jul 09 14:20:49.671468 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.671433 2563 generic.go:356] "Generic (PLEG): container finished" podID="3f7410766a9ef9617ac405e95724d1bd" containerID="2e01b5f1d66dc69b4475b0297a285b85947f043e5c668a2701f7b2fb3a2640fa" exitCode=0 Jul 09 14:20:49.671558 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.671518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" event={"ID":"3f7410766a9ef9617ac405e95724d1bd","Type":"ContainerDied","Data":"2e01b5f1d66dc69b4475b0297a285b85947f043e5c668a2701f7b2fb3a2640fa"} Jul 09 14:20:49.673109 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.673080 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" event={"ID":"5340419ec45aff54cbe78dced08dd234","Type":"ContainerStarted","Data":"c5137e8e0d4dcb714eae6c535d6e0c03aaae006a9743ae7eeb7530ee6b92e53a"} Jul 09 14:20:49.674682 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.674658 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sx47j" event={"ID":"d6a5d9db-5636-48cd-bb96-76b36e182c4b","Type":"ContainerStarted","Data":"b0236a27b3992ef414d22523a494066a0e55a8fe98f6341fb74951d10c3007ae"} Jul 09 14:20:49.676236 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.676216 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c393e5-2ad0-433a-9373-ff8aea28ed0c" containerID="df0f8359f155f0dd3bfae827012386d2f97d562c077c7b6a13367dabf545e79b" exitCode=0 Jul 09 14:20:49.676308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.676288 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerDied","Data":"df0f8359f155f0dd3bfae827012386d2f97d562c077c7b6a13367dabf545e79b"} Jul 09 14:20:49.677873 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.677853 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" event={"ID":"7c264538-ef22-42cf-bf5f-ec59ef08f27a","Type":"ContainerStarted","Data":"4c5c2fee325f969105fa0d3e7b827bfb5d698f9ad2fb717d795938dc7c5e7285"} Jul 09 14:20:49.679304 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.679216 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-95lfw" event={"ID":"abec23fe-04cb-43a0-917e-721e2f4e4ce0","Type":"ContainerStarted","Data":"a250a8aab599ab2196b2928bca775745b6e8666d7260aaaad85e32a3f783dca1"} Jul 09 14:20:49.684745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.684702 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b4jwm" podStartSLOduration=3.353660538 podStartE2EDuration="13.684691748s" totalImagesPullingTime="10.33103121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:49.684076537 +0000 UTC m=+13.705159821" watchObservedRunningTime="2026-07-09 14:20:49.684691748 +0000 UTC m=+13.705775012" Jul 09 14:20:49.696858 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.696814 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-55.ec2.internal" podStartSLOduration=12.696804038 podStartE2EDuration="12.696804038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:49.69655022 +0000 UTC m=+13.717633485" watchObservedRunningTime="2026-07-09 14:20:49.696804038 +0000 UTC m=+13.717887303" Jul 09 14:20:49.729366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.729319 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-95lfw" podStartSLOduration=3.318506338 podStartE2EDuration="13.729303302s" totalImagesPullingTime="10.410796964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:49.729081398 +0000 UTC m=+13.750164661" watchObservedRunningTime="2026-07-09 14:20:49.729303302 +0000 UTC m=+13.750386566" Jul 09 14:20:49.729893 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.729861 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zmmhc" podStartSLOduration=3.349290445 podStartE2EDuration="13.729851896s" totalImagesPullingTime="10.380561451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:49.710497352 +0000 UTC m=+13.731580618" watchObservedRunningTime="2026-07-09 14:20:49.729851896 +0000 UTC m=+13.750935163" Jul 09 14:20:49.771084 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:49.771038 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sx47j" podStartSLOduration=3.443932872 podStartE2EDuration="13.771020709s" totalImagesPullingTime="10.327087837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:49.770961224 +0000 UTC m=+13.792044516" watchObservedRunningTime="2026-07-09 14:20:49.771020709 +0000 UTC m=+13.792103973" Jul 09 14:20:50.569315 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:50.569289 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:50.569467 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:50.569426 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:50.683236 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:50.682991 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9868z" event={"ID":"8dd49727-e4e7-4de1-9765-e0c7f23be495","Type":"ContainerStarted","Data":"e149ad2b7956ca71ae9cf3e8492a9b55c59619fbe781220b3192b3e3393909fd"} Jul 09 14:20:50.685201 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:50.685128 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" event={"ID":"3f7410766a9ef9617ac405e95724d1bd","Type":"ContainerStarted","Data":"716f441a03a1463db2ab0bb6bc83fef572c95a2d6f3938cb148bf9080b8ed0f9"} Jul 09 14:20:50.700670 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:50.700604 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9868z" podStartSLOduration=4.381893172 podStartE2EDuration="14.70058514s" totalImagesPullingTime="10.318691968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:50.700455161 +0000 UTC m=+14.721538451" watchObservedRunningTime="2026-07-09 14:20:50.70058514 +0000 UTC m=+14.721668406" Jul 09 14:20:50.714732 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:50.714691 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-55.ec2.internal" podStartSLOduration=13.714655337 podStartE2EDuration="13.714655337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:20:50.71458746 +0000 UTC m=+14.735670726" watchObservedRunningTime="2026-07-09 14:20:50.714655337 +0000 UTC m=+14.735738600" Jul 09 14:20:51.523033 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:51.523002 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:51.523567 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:51.523546 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:20:51.569106 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:51.569083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:51.569220 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:51.569188 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:51.569395 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:51.569349 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:51.569531 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:51.569464 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:52.568912 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:52.568876 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:52.569300 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:52.569017 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:52.688211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:52.688185 2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 14:20:53.569296 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:53.569270 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:53.569296 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:53.569297 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:53.569927 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:53.569383 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:53.569927 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:53.569508 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:54.142268 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:54.142236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:54.142487 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:54.142287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:54.142487 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.142398 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:54.142487 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.142452 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs podName:4154366d-8c15-403c-8df9-cceea3424319 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:10.142438184 +0000 UTC m=+34.163521430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs") pod "network-metrics-daemon-fb2rh" (UID: "4154366d-8c15-403c-8df9-cceea3424319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:20:54.142487 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.142399 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:54.142700 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.142525 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret podName:e527178e-badd-4625-a67a-f5e2904fbc82 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:10.142513476 +0000 UTC m=+34.163596718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret") pod "global-pull-secret-syncer-928hb" (UID: "e527178e-badd-4625-a67a-f5e2904fbc82") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:20:54.242728 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:54.242701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:54.242863 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.242818 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:20:54.242863 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.242833 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:20:54.242863 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.242841 2563 projected.go:196] Error preparing data for projected volume kube-api-access-6vdnw for pod openshift-network-diagnostics/network-check-target-95fkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:54.242963 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.242886 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw podName:7a682c4e-eed0-4463-b5d3-b91351ebbe96 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:10.242872934 +0000 UTC m=+34.263956177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vdnw" (UniqueName: "kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw") pod "network-check-target-95fkk" (UID: "7a682c4e-eed0-4463-b5d3-b91351ebbe96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:20:54.569568 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:54.569540 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:54.570279 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:54.569694 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:55.568793 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:55.568759 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:55.568953 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:55.568879 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:55.569019 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:55.568967 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:55.569102 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:55.569074 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:56.570250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:56.570220 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:56.570768 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:56.570336 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:57.568867 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:57.568834 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:57.569036 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:57.568834 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:57.569036 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:57.568946 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:20:57.569036 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:57.569008 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:58.569151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:58.569126 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:20:58.569524 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:58.569258 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:20:59.568989 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:59.568958 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:20:59.569276 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:20:59.568958 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:20:59.569276 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:59.569059 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:20:59.569276 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:20:59.569131 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:00.569277 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:00.569244 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:00.569768 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:00.569381 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:01.247218 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.247056 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 14:21:01.461553 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.461484 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T14:21:01.247200278Z","UUID":"b7e33639-dfc9-4b6c-ba3a-08d661124a0b","Handler":null,"Name":"","Endpoint":""} Jul 09 14:21:01.462772 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.462757 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 14:21:01.462855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.462787 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 14:21:01.569140 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.569122 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:01.569225 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.569123 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:01.569225 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:01.569216 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:01.569293 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:01.569273 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:01.705171 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.705137 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c393e5-2ad0-433a-9373-ff8aea28ed0c" containerID="2c560ee0f8952e6811b47205511abbb7b6739c8533cc7f3d90df0fa8277a0873" exitCode=0 Jul 09 14:21:01.705553 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.705214 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerDied","Data":"2c560ee0f8952e6811b47205511abbb7b6739c8533cc7f3d90df0fa8277a0873"} Jul 09 14:21:01.709145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.709123 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k49pd" event={"ID":"df7474eb-d84f-475a-a445-21a49a350284","Type":"ContainerStarted","Data":"cba5482838be44fa011e7c9636469ceba87ec7ee66aa4b24536ce0129e16039f"} Jul 09 14:21:01.710913 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.710892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" event={"ID":"7c264538-ef22-42cf-bf5f-ec59ef08f27a","Type":"ContainerStarted","Data":"b77de1f99392628deeb8012a05e378cdbcc0fcb259592c2038c4019bafbba4c0"} Jul 09 14:21:01.713099 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.713081 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:21:01.713726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.713707 2563 generic.go:356] "Generic (PLEG): container finished" podID="b3aca0a7-2162-4481-bd9d-f2dee1a6fa33" containerID="e56572c58b84e661fed083cc8bb97237748980d3a0af160b5a9af57472c94440" exitCode=1 Jul 09 14:21:01.713826 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.713742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"1433d1d34ff2ae17b315202bbd49e511c58852d2e1bcc5cd20147e24af3926af"} Jul 09 14:21:01.713826 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.713785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"bf115ff798cc33b81fb2ee895d468ab266a949fec53004fc72a940408f222465"} Jul 09 14:21:01.713826 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.713818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"ffdb4d48ef5effcc960ad6d4712d68b2547f438bd6827c0ae16baa293ca0a59c"} Jul 09 14:21:01.713921 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.713850 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerDied","Data":"e56572c58b84e661fed083cc8bb97237748980d3a0af160b5a9af57472c94440"} Jul 09 14:21:01.713921 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.713878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"8264016a0edf073a2bff87efac16d306ad4683b4aaf0b758cfea72acf37c8328"} Jul 09 14:21:01.747337 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:01.747289 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k49pd" podStartSLOduration=3.495419727 podStartE2EDuration="25.747278346s" totalImagesPullingTime="22.251858619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:01.746378954 +0000 UTC m=+25.767462218" watchObservedRunningTime="2026-07-09 14:21:01.747278346 +0000 UTC m=+25.768361609" Jul 09 14:21:02.568977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:02.568827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:02.569102 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:02.569082 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:02.718035 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:02.718006 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" event={"ID":"7c264538-ef22-42cf-bf5f-ec59ef08f27a","Type":"ContainerStarted","Data":"9bda4f062fe963ea4ea1d48b6c4f24d1d12c0479e5ba0f081c6131f71951188c"} Jul 09 14:21:02.720695 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:02.720675 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:21:02.721073 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:02.721050 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"d0abbf9756d42e7024ede8e8711b0ee89350b18febd58a8f44831e13b1010c61"} Jul 09 14:21:02.735735 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:02.735698 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ffp5k" podStartSLOduration=3.5400945889999997 podStartE2EDuration="26.735686289s" totalImagesPullingTime="23.1955917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:02.735679464 +0000 UTC m=+26.756762747" watchObservedRunningTime="2026-07-09 14:21:02.735686289 +0000 UTC m=+26.756769547" Jul 09 14:21:03.406103 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:03.406074 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:21:03.406326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:03.406171 2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 14:21:03.406678 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:03.406657 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmmhc" Jul 09 14:21:03.569637 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:03.569599 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:03.569774 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:03.569690 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:03.569774 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:03.569604 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:03.569774 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:03.569748 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:04.569522 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:04.569487 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:04.570128 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:04.569599 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:04.727844 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:04.727814 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c393e5-2ad0-433a-9373-ff8aea28ed0c" containerID="4bd305ec84d8364d1ac4ce96fbbb5748f0e6e05ace7b00d64b3fe0b118d39833" exitCode=0 Jul 09 14:21:04.727985 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:04.727914 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerDied","Data":"4bd305ec84d8364d1ac4ce96fbbb5748f0e6e05ace7b00d64b3fe0b118d39833"} Jul 09 14:21:04.731653 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:04.731635 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:21:04.732054 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:04.732033 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"dbaa7e3e73677eef1856d382179c08f18ad41aad2e1f8806bb0e43c61abfbf87"} Jul 09 14:21:05.569663 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:05.569635 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:05.570077 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:05.569723 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:05.570077 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:05.569778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:05.570077 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:05.569854 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:06.569610 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.569577 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:06.569772 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:06.569676 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:06.740180 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.740157 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:21:06.740549 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.740524 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"3f67d7ef4fef87f34527c61cb5aa23a16be9eefc309d0f1983af900e4274edd7"} Jul 09 14:21:06.740774 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.740740 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:21:06.740953 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.740936 2563 scope.go:117] "RemoveContainer" containerID="e56572c58b84e661fed083cc8bb97237748980d3a0af160b5a9af57472c94440" Jul 09 14:21:06.743501 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.743476 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c393e5-2ad0-433a-9373-ff8aea28ed0c" containerID="265528985b4acb1c59b82f07cd3928a55ec342dc2a30e79947166bd09417ac3a" exitCode=0 Jul 09 14:21:06.743553 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.743518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerDied","Data":"265528985b4acb1c59b82f07cd3928a55ec342dc2a30e79947166bd09417ac3a"} Jul 09 14:21:06.757988 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:06.757969 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:21:07.569343 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.569319 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:07.569491 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.569320 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:07.569491 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:07.569410 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:07.569592 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:07.569511 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:07.752668 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.752639 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:21:07.753381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.753046 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" event={"ID":"b3aca0a7-2162-4481-bd9d-f2dee1a6fa33","Type":"ContainerStarted","Data":"2b7c27847fd8fa4d064d98b2349cf67d41526b5ff1268e614c53c80567c67f07"} Jul 09 14:21:07.753717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.753695 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:21:07.753820 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.753771 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:21:07.772157 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.772111 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:21:07.783913 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.783867 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" podStartSLOduration=9.509452225 podStartE2EDuration="31.783848713s" totalImagesPullingTime="22.274396488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:07.78268119 +0000 UTC m=+31.803764477" watchObservedRunningTime="2026-07-09 14:21:07.783848713 +0000 UTC m=+31.804931979" Jul 09 14:21:07.998107 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.998030 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95fkk"] Jul 09 14:21:07.998361 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.998213 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:07.998361 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:07.998311 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:07.998783 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.998764 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-928hb"] Jul 09 14:21:07.998869 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:07.998857 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:07.998963 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:07.998945 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:08.000471 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:08.000439 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fb2rh"] Jul 09 14:21:08.000555 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:08.000525 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:08.000684 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:08.000613 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:09.569342 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:09.569301 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:09.569864 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:09.569414 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:09.569864 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:09.569423 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:09.569864 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:09.569448 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:09.569864 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:09.569533 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:09.569864 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:09.569611 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:10.167904 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:10.167877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:10.168028 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:10.167920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:10.168070 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.168034 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 14:21:10.168070 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.168038 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:21:10.168153 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.168086 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret podName:e527178e-badd-4625-a67a-f5e2904fbc82 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:42.168070248 +0000 UTC m=+66.189153497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret") pod "global-pull-secret-syncer-928hb" (UID: "e527178e-badd-4625-a67a-f5e2904fbc82") : object "kube-system"/"original-pull-secret" not registered Jul 09 14:21:10.168153 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.168100 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs podName:4154366d-8c15-403c-8df9-cceea3424319 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:42.168093683 +0000 UTC m=+66.189176926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs") pod "network-metrics-daemon-fb2rh" (UID: "4154366d-8c15-403c-8df9-cceea3424319") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 14:21:10.269160 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:10.269130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:10.269281 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.269245 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 14:21:10.269281 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.269260 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 14:21:10.269281 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.269269 2563 projected.go:196] Error preparing data for projected volume kube-api-access-6vdnw for pod openshift-network-diagnostics/network-check-target-95fkk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:21:10.269403 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:10.269315 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw podName:7a682c4e-eed0-4463-b5d3-b91351ebbe96 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:42.269299555 +0000 UTC m=+66.290382799 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6vdnw" (UniqueName: "kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw") pod "network-check-target-95fkk" (UID: "7a682c4e-eed0-4463-b5d3-b91351ebbe96") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 14:21:11.568938 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:11.568875 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:11.569364 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:11.568996 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:11.569364 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:11.569005 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:11.569364 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:11.569031 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:11.569364 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:11.569127 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:11.569364 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:11.569211 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:12.769011 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:12.768802 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c393e5-2ad0-433a-9373-ff8aea28ed0c" containerID="fdc8c32d3e77403cd8a812faa5b14ed7c984fd78fb81284a0cd390307d237233" exitCode=0 Jul 09 14:21:12.769011 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:12.768887 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerDied","Data":"fdc8c32d3e77403cd8a812faa5b14ed7c984fd78fb81284a0cd390307d237233"} Jul 09 14:21:13.569584 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.569560 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:13.569703 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.569561 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:13.569703 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:13.569672 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-95fkk" podUID="7a682c4e-eed0-4463-b5d3-b91351ebbe96" Jul 09 14:21:13.569766 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.569565 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:13.569766 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:13.569744 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fb2rh" podUID="4154366d-8c15-403c-8df9-cceea3424319" Jul 09 14:21:13.569819 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:13.569804 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-928hb" podUID="e527178e-badd-4625-a67a-f5e2904fbc82" Jul 09 14:21:13.734654 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.734580 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-55.ec2.internal" event="NodeReady" Jul 09 14:21:13.734764 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.734693 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 14:21:13.774634 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.774587 2563 generic.go:356] "Generic (PLEG): container finished" podID="c5c393e5-2ad0-433a-9373-ff8aea28ed0c" containerID="d39621cb810b940725f880657e698e39f15b16cb13170fc9c1b7a8663f7fb21a" exitCode=0 Jul 09 14:21:13.775162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.774729 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerDied","Data":"d39621cb810b940725f880657e698e39f15b16cb13170fc9c1b7a8663f7fb21a"} Jul 09 14:21:13.776143 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.776122 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74d4c9b7c8-5tm8d"] Jul 09 14:21:13.777966 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.777949 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.780750 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.780728 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 14:21:13.780851 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.780760 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 14:21:13.780959 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.780936 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 14:21:13.781041 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.781027 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-khbqc\"" Jul 09 14:21:13.791995 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.791966 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b"] Jul 09 14:21:13.794145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.794114 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 14:21:13.795878 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.794925 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw"] Jul 09 14:21:13.795878 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.795195 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:13.797040 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.797003 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-fwnfg"] Jul 09 14:21:13.797258 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.797057 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.798607 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.797883 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8fl5v\"" Jul 09 14:21:13.800839 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.800818 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr"] Jul 09 14:21:13.801862 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.801818 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 14:21:13.802216 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.802083 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.802375 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.802355 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 14:21:13.802814 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.802696 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vg5pq\"" Jul 09 14:21:13.802929 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.802838 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:13.803015 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.802939 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.803083 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.803018 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm"] Jul 09 14:21:13.803133 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.803124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 14:21:13.804000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.803304 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 14:21:13.804000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.803506 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.804545 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.804449 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.804880 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.804828 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" Jul 09 14:21:13.808696 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.807174 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl"] Jul 09 14:21:13.808696 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.808126 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:13.812922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.812894 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74d4c9b7c8-5tm8d"] Jul 09 14:21:13.813066 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.813053 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2fwcm"] Jul 09 14:21:13.813777 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.813249 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:13.819591 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.819569 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.819882 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.819862 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.820432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.819998 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.820432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.820092 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.820432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.820110 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.820432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.820225 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 14:21:13.820432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.820240 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 14:21:13.820432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.820384 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 14:21:13.820432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.820414 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 14:21:13.821481 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.821452 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw"] Jul 09 14:21:13.821573 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.821485 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5bdf54bb-m5z9v"] Jul 09 14:21:13.822486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.822155 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x5fqd\"" Jul 09 14:21:13.822486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.822329 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:13.822486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.822380 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.822853 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.822773 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.822853 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.822803 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-n7b7b\"" Jul 09 14:21:13.822995 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.822930 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mvsg2\"" Jul 09 14:21:13.822995 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.822956 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kv6gx\"" Jul 09 14:21:13.823084 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.823060 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.823633 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.823600 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 14:21:13.826013 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.825991 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b"] Jul 09 14:21:13.826393 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.826045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-fwnfg"] Jul 09 14:21:13.826393 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.826069 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr"] Jul 09 14:21:13.826393 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.826092 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-bbxxs"] Jul 09 14:21:13.826916 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.826899 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:13.832067 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.832041 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 14:21:13.832466 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.832430 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 14:21:13.835681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.833031 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.835681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.833741 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 14:21:13.835681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.834120 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 14:21:13.835681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.834461 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.835681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.834841 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxvxp\"" Jul 09 14:21:13.835681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.835442 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x8gpr\"" Jul 09 14:21:13.836033 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.835817 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 14:21:13.836033 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.835869 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 14:21:13.836253 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.836212 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wwjv6"] Jul 09 14:21:13.837954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.837923 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:13.840987 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.840966 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq"] Jul 09 14:21:13.841105 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.841067 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:13.842551 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.842530 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 14:21:13.842871 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.842852 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 14:21:13.842988 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.842972 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vj47m\"" Jul 09 14:21:13.843238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.843210 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.843457 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.843438 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.843847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.843828 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 14:21:13.844989 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.844972 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w8kmw\"" Jul 09 14:21:13.845218 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.845141 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 14:21:13.845527 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.845506 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fwcm"] Jul 09 14:21:13.845608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.845550 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bdf54bb-m5z9v"] Jul 09 14:21:13.845608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.845568 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl"] Jul 09 14:21:13.845720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.845607 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 14:21:13.845720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.845661 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" Jul 09 14:21:13.848159 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.848092 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wwjv6"] Jul 09 14:21:13.849995 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.849975 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.850138 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.850020 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.850237 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.850220 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dfz8j\"" Jul 09 14:21:13.850470 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.850453 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-bbxxs"] Jul 09 14:21:13.851083 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.851064 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm"] Jul 09 14:21:13.853787 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.853744 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 14:21:13.872062 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.872046 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq"] Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894437 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1858e3a2-d403-45fc-88d3-d18aca4057c6-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-installation-pull-secrets\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mx9\" (UniqueName: \"kubernetes.io/projected/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-kube-api-access-q8mx9\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894633 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e2a0f12-5b5a-47b2-8539-208a59706e7c-ca-trust-extracted\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894659 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-config\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894758 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-image-registry-private-configuration\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894806 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br488\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-kube-api-access-br488\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894841 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgm5\" (UniqueName: \"kubernetes.io/projected/1858e3a2-d403-45fc-88d3-d18aca4057c6-kube-api-access-fwgm5\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-bound-sa-token\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.895721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894888 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-certificates\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.896230 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-trusted-ca\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.896230 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.894944 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1858e3a2-d403-45fc-88d3-d18aca4057c6-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:13.900696 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.900677 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-27qjw"] Jul 09 14:21:13.908301 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.908286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:13.910936 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.910920 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 14:21:13.911031 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.910998 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 14:21:13.911327 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.911309 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 14:21:13.911479 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.911458 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n5vbv\"" Jul 09 14:21:13.914487 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.914466 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-27qjw"] Jul 09 14:21:13.995459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e2a0f12-5b5a-47b2-8539-208a59706e7c-ca-trust-extracted\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.995459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995440 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-default-certificate\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:13.995459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x446\" (UniqueName: \"kubernetes.io/projected/5a553e49-b8e4-4543-9550-9eba5165cc3a-kube-api-access-9x446\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:13.995594 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1858e3a2-d403-45fc-88d3-d18aca4057c6-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:13.995594 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995545 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-config\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.995594 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995579 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:13.995726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995633 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs52p\" (UniqueName: \"kubernetes.io/projected/1329b344-4e47-4d7a-b4c5-00fd4a60435e-kube-api-access-vs52p\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:13.995726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6966a58b-4631-44d0-8087-1bf9f97bdab3-trusted-ca\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:13.995726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995690 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-tmp\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:13.995726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:13.995874 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995730 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:13.995874 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:13.995874 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995823 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-image-registry-private-configuration\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.995874 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995835 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e2a0f12-5b5a-47b2-8539-208a59706e7c-ca-trust-extracted\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.995874 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br488\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-kube-api-access-br488\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqhjc\" (UniqueName: \"kubernetes.io/projected/7761ed1a-569d-497c-8f38-af325db1a0cf-kube-api-access-mqhjc\") pod \"volume-data-source-validator-78495597f6-gm4nr\" (UID: \"7761ed1a-569d-497c-8f38-af325db1a0cf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lrf\" (UniqueName: \"kubernetes.io/projected/e5586c00-c5ed-458c-8001-abf514dda47b-kube-api-access-s2lrf\") pod \"network-check-source-869dcfcdcd-m5nnq\" (UID: \"e5586c00-c5ed-458c-8001-abf514dda47b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1329b344-4e47-4d7a-b4c5-00fd4a60435e-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995945 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-bound-sa-token\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-certificates\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.995983 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-trusted-ca\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996004 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-snapshots\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-installation-pull-secrets\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.996144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996091 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-config\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996166 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8g8\" (UniqueName: \"kubernetes.io/projected/4167d24e-780d-478f-938b-6efe2af7bdb5-kube-api-access-2j8g8\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996203 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48z48\" (UniqueName: \"kubernetes.io/projected/91ad122e-b32c-483b-89c3-fac94cbfc115-kube-api-access-48z48\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mx9\" (UniqueName: \"kubernetes.io/projected/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-kube-api-access-q8mx9\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad122e-b32c-483b-89c3-fac94cbfc115-config-volume\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996400 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-stats-auth\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996428 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmz6\" (UniqueName: \"kubernetes.io/projected/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-kube-api-access-szmz6\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996541 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996564 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:13.996599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6966a58b-4631-44d0-8087-1bf9f97bdab3-config\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9cm\" (UniqueName: \"kubernetes.io/projected/6966a58b-4631-44d0-8087-1bf9f97bdab3-kube-api-access-hb9cm\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:13.996658 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:13.996673 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74d4c9b7c8-5tm8d: secret "image-registry-tls" not found Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996654 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-certificates\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996660 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ad122e-b32c-483b-89c3-fac94cbfc115-tmp-dir\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:13.996729 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls podName:4e2a0f12-5b5a-47b2-8539-208a59706e7c nodeName:}" failed. No retries permitted until 2026-07-09 14:21:14.496713402 +0000 UTC m=+38.517796660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls") pod "image-registry-74d4c9b7c8-5tm8d" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c") : secret "image-registry-tls" not found Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996760 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-serving-cert\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6966a58b-4631-44d0-8087-1bf9f97bdab3-serving-cert\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996830 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgm5\" (UniqueName: \"kubernetes.io/projected/1858e3a2-d403-45fc-88d3-d18aca4057c6-kube-api-access-fwgm5\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1858e3a2-d403-45fc-88d3-d18aca4057c6-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:13.997088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996912 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:13.997892 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.996951 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:13.997892 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.997083 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-trusted-ca\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:13.998351 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:13.998332 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1858e3a2-d403-45fc-88d3-d18aca4057c6-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:14.000306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.000278 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1858e3a2-d403-45fc-88d3-d18aca4057c6-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:14.000306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.000302 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:14.000454 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.000400 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-image-registry-private-configuration\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:14.000454 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.000442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-installation-pull-secrets\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:14.007778 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.007363 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br488\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-kube-api-access-br488\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:14.007778 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.007744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-bound-sa-token\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:14.008224 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.008205 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgm5\" (UniqueName: \"kubernetes.io/projected/1858e3a2-d403-45fc-88d3-d18aca4057c6-kube-api-access-fwgm5\") pod \"kube-storage-version-migrator-operator-799cb7f958-88h2b\" (UID: \"1858e3a2-d403-45fc-88d3-d18aca4057c6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:14.008630 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.008594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mx9\" (UniqueName: \"kubernetes.io/projected/12ecb533-eadf-4be8-86f8-b8a843f0fd7e-kube-api-access-q8mx9\") pod \"service-ca-operator-7695fc6c8c-2fgcw\" (UID: \"12ecb533-eadf-4be8-86f8-b8a843f0fd7e\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:14.097738 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.097717 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8g8\" (UniqueName: \"kubernetes.io/projected/4167d24e-780d-478f-938b-6efe2af7bdb5-kube-api-access-2j8g8\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:14.097816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.097744 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48z48\" (UniqueName: \"kubernetes.io/projected/91ad122e-b32c-483b-89c3-fac94cbfc115-kube-api-access-48z48\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.097816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.097762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad122e-b32c-483b-89c3-fac94cbfc115-config-volume\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.097816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.097775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-stats-auth\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.097816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.097792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmz6\" (UniqueName: \"kubernetes.io/projected/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-kube-api-access-szmz6\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.098014 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.097919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.098014 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.097975 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:14.098014 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6966a58b-4631-44d0-8087-1bf9f97bdab3-config\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.098151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9cm\" (UniqueName: \"kubernetes.io/projected/6966a58b-4631-44d0-8087-1bf9f97bdab3-kube-api-access-hb9cm\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.098151 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.098046 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:14.098151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ad122e-b32c-483b-89c3-fac94cbfc115-tmp-dir\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.098151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-serving-cert\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.098151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6966a58b-4631-44d0-8087-1bf9f97bdab3-serving-cert\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.098151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.098387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9889133b-8a20-4733-a2ed-ea4f39c13ece-cert\") pod \"ingress-canary-27qjw\" (UID: \"9889133b-8a20-4733-a2ed-ea4f39c13ece\") " pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:14.098387 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.098210 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls podName:1329b344-4e47-4d7a-b4c5-00fd4a60435e nodeName:}" failed. No retries permitted until 2026-07-09 14:21:14.598191344 +0000 UTC m=+38.619274593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4zmfm" (UID: "1329b344-4e47-4d7a-b4c5-00fd4a60435e") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:14.098541 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098524 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ad122e-b32c-483b-89c3-fac94cbfc115-tmp-dir\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.098596 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.098670 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098645 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpg59\" (UniqueName: \"kubernetes.io/projected/9889133b-8a20-4733-a2ed-ea4f39c13ece-kube-api-access-jpg59\") pod \"ingress-canary-27qjw\" (UID: \"9889133b-8a20-4733-a2ed-ea4f39c13ece\") " pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:14.098717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098665 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.098717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098674 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.098717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-default-certificate\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.098865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x446\" (UniqueName: \"kubernetes.io/projected/5a553e49-b8e4-4543-9550-9eba5165cc3a-kube-api-access-9x446\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.098865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:14.098865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098766 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6966a58b-4631-44d0-8087-1bf9f97bdab3-config\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.098865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs52p\" (UniqueName: \"kubernetes.io/projected/1329b344-4e47-4d7a-b4c5-00fd4a60435e-kube-api-access-vs52p\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:14.098865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6966a58b-4631-44d0-8087-1bf9f97bdab3-trusted-ca\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.098865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098838 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-tmp\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.098865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:14.099177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.099177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:14.099177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098946 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqhjc\" (UniqueName: \"kubernetes.io/projected/7761ed1a-569d-497c-8f38-af325db1a0cf-kube-api-access-mqhjc\") pod \"volume-data-source-validator-78495597f6-gm4nr\" (UID: \"7761ed1a-569d-497c-8f38-af325db1a0cf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" Jul 09 14:21:14.099177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098974 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lrf\" (UniqueName: \"kubernetes.io/projected/e5586c00-c5ed-458c-8001-abf514dda47b-kube-api-access-s2lrf\") pod \"network-check-source-869dcfcdcd-m5nnq\" (UID: \"e5586c00-c5ed-458c-8001-abf514dda47b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" Jul 09 14:21:14.099177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.098999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1329b344-4e47-4d7a-b4c5-00fd4a60435e-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:14.099177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.099064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-snapshots\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.099177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.099126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.099494 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.098769 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:21:14.099494 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.098852 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:21:14.099494 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.099430 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls podName:91ad122e-b32c-483b-89c3-fac94cbfc115 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:14.599411542 +0000 UTC m=+38.620494810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls") pod "dns-default-2fwcm" (UID: "91ad122e-b32c-483b-89c3-fac94cbfc115") : secret "dns-default-metrics-tls" not found Jul 09 14:21:14.099494 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.099453 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:14.59944256 +0000 UTC m=+38.620525806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : configmap references non-existent config key: service-ca.crt Jul 09 14:21:14.099705 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.099578 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert podName:25e5eacf-4b28-4bdb-aa9e-214257dfe58a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:14.599566461 +0000 UTC m=+38.620649704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wwjv6" (UID: "25e5eacf-4b28-4bdb-aa9e-214257dfe58a") : secret "networking-console-plugin-cert" not found Jul 09 14:21:14.099705 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.099583 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-snapshots\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.099705 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.099656 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:21:14.099705 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.099691 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls podName:4167d24e-780d-478f-938b-6efe2af7bdb5 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:14.599680456 +0000 UTC m=+38.620763699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls") pod "cluster-samples-operator-65db858894-rfbpl" (UID: "4167d24e-780d-478f-938b-6efe2af7bdb5") : secret "samples-operator-tls" not found Jul 09 14:21:14.099905 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.099818 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:21:14.099956 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.099922 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:14.599907097 +0000 UTC m=+38.620990356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : secret "router-metrics-certs-default" not found Jul 09 14:21:14.100161 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.100106 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1329b344-4e47-4d7a-b4c5-00fd4a60435e-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:14.100255 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.100163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:14.100314 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.100280 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-tmp\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.100314 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.100301 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6966a58b-4631-44d0-8087-1bf9f97bdab3-trusted-ca\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.100411 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.100365 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-stats-auth\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.100638 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.100605 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-serving-cert\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.101070 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.101054 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-default-certificate\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.101134 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.101056 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6966a58b-4631-44d0-8087-1bf9f97bdab3-serving-cert\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.102305 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.102286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/91ad122e-b32c-483b-89c3-fac94cbfc115-config-volume\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.107173 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.107129 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmz6\" (UniqueName: \"kubernetes.io/projected/7ae68662-8dc0-48d2-ad61-b6e631f5fc81-kube-api-access-szmz6\") pod \"insights-operator-68d9dd94cf-fwnfg\" (UID: \"7ae68662-8dc0-48d2-ad61-b6e631f5fc81\") " pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.107257 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.107236 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48z48\" (UniqueName: \"kubernetes.io/projected/91ad122e-b32c-483b-89c3-fac94cbfc115-kube-api-access-48z48\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.108005 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.107983 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9cm\" (UniqueName: \"kubernetes.io/projected/6966a58b-4631-44d0-8087-1bf9f97bdab3-kube-api-access-hb9cm\") pod \"console-operator-7866497494-bbxxs\" (UID: \"6966a58b-4631-44d0-8087-1bf9f97bdab3\") " pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.108161 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.108147 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8g8\" (UniqueName: \"kubernetes.io/projected/4167d24e-780d-478f-938b-6efe2af7bdb5-kube-api-access-2j8g8\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:14.109377 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.109353 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lrf\" (UniqueName: \"kubernetes.io/projected/e5586c00-c5ed-458c-8001-abf514dda47b-kube-api-access-s2lrf\") pod \"network-check-source-869dcfcdcd-m5nnq\" (UID: \"e5586c00-c5ed-458c-8001-abf514dda47b\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" Jul 09 14:21:14.109490 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.109460 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs52p\" (UniqueName: \"kubernetes.io/projected/1329b344-4e47-4d7a-b4c5-00fd4a60435e-kube-api-access-vs52p\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:14.109548 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.109519 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" Jul 09 14:21:14.110602 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.110583 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqhjc\" (UniqueName: \"kubernetes.io/projected/7761ed1a-569d-497c-8f38-af325db1a0cf-kube-api-access-mqhjc\") pod \"volume-data-source-validator-78495597f6-gm4nr\" (UID: \"7761ed1a-569d-497c-8f38-af325db1a0cf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" Jul 09 14:21:14.110680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.110655 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x446\" (UniqueName: \"kubernetes.io/projected/5a553e49-b8e4-4543-9550-9eba5165cc3a-kube-api-access-9x446\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.121266 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.121249 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" Jul 09 14:21:14.131247 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.131231 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" Jul 09 14:21:14.142208 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.142193 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" Jul 09 14:21:14.200953 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.200397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9889133b-8a20-4733-a2ed-ea4f39c13ece-cert\") pod \"ingress-canary-27qjw\" (UID: \"9889133b-8a20-4733-a2ed-ea4f39c13ece\") " pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:14.200953 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.200455 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpg59\" (UniqueName: \"kubernetes.io/projected/9889133b-8a20-4733-a2ed-ea4f39c13ece-kube-api-access-jpg59\") pod \"ingress-canary-27qjw\" (UID: \"9889133b-8a20-4733-a2ed-ea4f39c13ece\") " pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:14.202761 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.202512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:14.204361 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.204141 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9889133b-8a20-4733-a2ed-ea4f39c13ece-cert\") pod \"ingress-canary-27qjw\" (UID: \"9889133b-8a20-4733-a2ed-ea4f39c13ece\") " pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:14.211714 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.209755 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" Jul 09 14:21:14.213294 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.213244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpg59\" (UniqueName: \"kubernetes.io/projected/9889133b-8a20-4733-a2ed-ea4f39c13ece-kube-api-access-jpg59\") pod \"ingress-canary-27qjw\" (UID: \"9889133b-8a20-4733-a2ed-ea4f39c13ece\") " pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:14.215900 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.215844 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-27qjw" Jul 09 14:21:14.348407 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.347847 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw"] Jul 09 14:21:14.349813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.349307 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-fwnfg"] Jul 09 14:21:14.350042 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.350012 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b"] Jul 09 14:21:14.351633 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:14.351576 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1858e3a2_d403_45fc_88d3_d18aca4057c6.slice/crio-144b1913c529ded23c901c27ddd23de15feb7e27da51e1940385485b38472e7b WatchSource:0}: Error finding container 144b1913c529ded23c901c27ddd23de15feb7e27da51e1940385485b38472e7b: Status 404 returned error can't find the container with id 144b1913c529ded23c901c27ddd23de15feb7e27da51e1940385485b38472e7b Jul 09 14:21:14.352170 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:14.352140 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae68662_8dc0_48d2_ad61_b6e631f5fc81.slice/crio-cb0d993c0c1298617c0d37be46445fc95893dad8ad664d068df9114775c59b95 WatchSource:0}: Error finding container cb0d993c0c1298617c0d37be46445fc95893dad8ad664d068df9114775c59b95: Status 404 returned error can't find the container with id cb0d993c0c1298617c0d37be46445fc95893dad8ad664d068df9114775c59b95 Jul 09 14:21:14.354407 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:14.354306 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ecb533_eadf_4be8_86f8_b8a843f0fd7e.slice/crio-6d87fe3fa9c9acd6720f2e867a00f5a0c784b7163e667700505be2af7eec6876 WatchSource:0}: Error finding container 6d87fe3fa9c9acd6720f2e867a00f5a0c784b7163e667700505be2af7eec6876: Status 404 returned error can't find the container with id 6d87fe3fa9c9acd6720f2e867a00f5a0c784b7163e667700505be2af7eec6876 Jul 09 14:21:14.356700 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.354780 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr"] Jul 09 14:21:14.359044 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:14.359021 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7761ed1a_569d_497c_8f38_af325db1a0cf.slice/crio-d3cf59607f25cb911c4105b0fa646bdd409c5e3d36218c31b995a560b0e5bb3e WatchSource:0}: Error finding container d3cf59607f25cb911c4105b0fa646bdd409c5e3d36218c31b995a560b0e5bb3e: Status 404 returned error can't find the container with id d3cf59607f25cb911c4105b0fa646bdd409c5e3d36218c31b995a560b0e5bb3e Jul 09 14:21:14.410267 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.410240 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-27qjw"] Jul 09 14:21:14.412884 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:14.412842 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9889133b_8a20_4733_a2ed_ea4f39c13ece.slice/crio-76ef7788f1ac4f37d3cc3fb3a6fb3e2119b6b2c427860aeb52a6f21af7dae81f WatchSource:0}: Error finding container 76ef7788f1ac4f37d3cc3fb3a6fb3e2119b6b2c427860aeb52a6f21af7dae81f: Status 404 returned error can't find the container with id 76ef7788f1ac4f37d3cc3fb3a6fb3e2119b6b2c427860aeb52a6f21af7dae81f Jul 09 14:21:14.415017 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.414994 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-bbxxs"] Jul 09 14:21:14.415962 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.415944 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq"] Jul 09 14:21:14.416661 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:14.416639 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5586c00_c5ed_458c_8001_abf514dda47b.slice/crio-43bd3a06aaf60b5f20e346dcce247e143a87c573865ee5e2fc7fff0b198c4ecb WatchSource:0}: Error finding container 43bd3a06aaf60b5f20e346dcce247e143a87c573865ee5e2fc7fff0b198c4ecb: Status 404 returned error can't find the container with id 43bd3a06aaf60b5f20e346dcce247e143a87c573865ee5e2fc7fff0b198c4ecb Jul 09 14:21:14.423524 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:14.423498 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6966a58b_4631_44d0_8087_1bf9f97bdab3.slice/crio-0a2f78d4386bc6aaa998f08dfa1c2f60b961f26ee5a12390f1ee66d86281c2d8 WatchSource:0}: Error finding container 0a2f78d4386bc6aaa998f08dfa1c2f60b961f26ee5a12390f1ee66d86281c2d8: Status 404 returned error can't find the container with id 0a2f78d4386bc6aaa998f08dfa1c2f60b961f26ee5a12390f1ee66d86281c2d8 Jul 09 14:21:14.504121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.504073 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:14.504217 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.504201 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:21:14.504255 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.504219 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74d4c9b7c8-5tm8d: secret "image-registry-tls" not found Jul 09 14:21:14.504326 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.504262 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls podName:4e2a0f12-5b5a-47b2-8539-208a59706e7c nodeName:}" failed. No retries permitted until 2026-07-09 14:21:15.504247048 +0000 UTC m=+39.525330318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls") pod "image-registry-74d4c9b7c8-5tm8d" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c") : secret "image-registry-tls" not found Jul 09 14:21:14.604718 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.604691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:14.604801 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.604746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:14.604801 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.604782 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.604869 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.604817 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:14.604869 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604824 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:14.604934 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604871 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls podName:1329b344-4e47-4d7a-b4c5-00fd4a60435e nodeName:}" failed. No retries permitted until 2026-07-09 14:21:15.604856121 +0000 UTC m=+39.625939383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4zmfm" (UID: "1329b344-4e47-4d7a-b4c5-00fd4a60435e") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:14.604934 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.604889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:14.604934 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604893 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:21:14.604934 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604912 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:15.604900515 +0000 UTC m=+39.625983757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : configmap references non-existent config key: service-ca.crt Jul 09 14:21:14.604934 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604895 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:21:14.604934 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604935 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls podName:91ad122e-b32c-483b-89c3-fac94cbfc115 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:15.604929113 +0000 UTC m=+39.626012355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls") pod "dns-default-2fwcm" (UID: "91ad122e-b32c-483b-89c3-fac94cbfc115") : secret "dns-default-metrics-tls" not found Jul 09 14:21:14.605122 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604938 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:21:14.605122 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604962 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:15.604954822 +0000 UTC m=+39.626038065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : secret "router-metrics-certs-default" not found Jul 09 14:21:14.605122 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.604961 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:14.605122 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.604972 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert podName:25e5eacf-4b28-4bdb-aa9e-214257dfe58a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:15.604966753 +0000 UTC m=+39.626049996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wwjv6" (UID: "25e5eacf-4b28-4bdb-aa9e-214257dfe58a") : secret "networking-console-plugin-cert" not found Jul 09 14:21:14.605122 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.605005 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:21:14.605122 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:14.605033 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls podName:4167d24e-780d-478f-938b-6efe2af7bdb5 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:15.605023047 +0000 UTC m=+39.626106290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls") pod "cluster-samples-operator-65db858894-rfbpl" (UID: "4167d24e-780d-478f-938b-6efe2af7bdb5") : secret "samples-operator-tls" not found Jul 09 14:21:14.778151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.778119 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" event={"ID":"7761ed1a-569d-497c-8f38-af325db1a0cf","Type":"ContainerStarted","Data":"d3cf59607f25cb911c4105b0fa646bdd409c5e3d36218c31b995a560b0e5bb3e"} Jul 09 14:21:14.779170 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.779145 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-27qjw" event={"ID":"9889133b-8a20-4733-a2ed-ea4f39c13ece","Type":"ContainerStarted","Data":"76ef7788f1ac4f37d3cc3fb3a6fb3e2119b6b2c427860aeb52a6f21af7dae81f"} Jul 09 14:21:14.780260 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.780240 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-bbxxs" event={"ID":"6966a58b-4631-44d0-8087-1bf9f97bdab3","Type":"ContainerStarted","Data":"0a2f78d4386bc6aaa998f08dfa1c2f60b961f26ee5a12390f1ee66d86281c2d8"} Jul 09 14:21:14.781256 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.781228 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" event={"ID":"1858e3a2-d403-45fc-88d3-d18aca4057c6","Type":"ContainerStarted","Data":"144b1913c529ded23c901c27ddd23de15feb7e27da51e1940385485b38472e7b"} Jul 09 14:21:14.784487 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.784469 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" event={"ID":"c5c393e5-2ad0-433a-9373-ff8aea28ed0c","Type":"ContainerStarted","Data":"c398aaea74185ce55453539712e1237231121a0ad7663ecb63c63c16193e45e7"} Jul 09 14:21:14.785558 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.785540 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" event={"ID":"e5586c00-c5ed-458c-8001-abf514dda47b","Type":"ContainerStarted","Data":"43bd3a06aaf60b5f20e346dcce247e143a87c573865ee5e2fc7fff0b198c4ecb"} Jul 09 14:21:14.786524 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.786499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" event={"ID":"12ecb533-eadf-4be8-86f8-b8a843f0fd7e","Type":"ContainerStarted","Data":"6d87fe3fa9c9acd6720f2e867a00f5a0c784b7163e667700505be2af7eec6876"} Jul 09 14:21:14.787492 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.787476 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" event={"ID":"7ae68662-8dc0-48d2-ad61-b6e631f5fc81","Type":"ContainerStarted","Data":"cb0d993c0c1298617c0d37be46445fc95893dad8ad664d068df9114775c59b95"} Jul 09 14:21:14.808887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:14.808851 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jnrwr" podStartSLOduration=8.934882063 podStartE2EDuration="38.808838529s" totalImagesPullingTime="29.873956466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:14.807155721 +0000 UTC m=+38.828238985" watchObservedRunningTime="2026-07-09 14:21:14.808838529 +0000 UTC m=+38.829921794" Jul 09 14:21:15.514466 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.514073 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:15.514466 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.514308 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:21:15.514466 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.514325 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74d4c9b7c8-5tm8d: secret "image-registry-tls" not found Jul 09 14:21:15.514466 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.514382 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls podName:4e2a0f12-5b5a-47b2-8539-208a59706e7c nodeName:}" failed. No retries permitted until 2026-07-09 14:21:17.514365185 +0000 UTC m=+41.535448434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls") pod "image-registry-74d4c9b7c8-5tm8d" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c") : secret "image-registry-tls" not found Jul 09 14:21:15.570104 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.568976 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:15.570104 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.569445 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:15.570104 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.569862 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:15.574659 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.573681 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bcf26\"" Jul 09 14:21:15.574659 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.573923 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 14:21:15.574659 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.574108 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4gsp\"" Jul 09 14:21:15.574659 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.574269 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.615267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.615327 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.615379 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.615417 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.615450 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:15.615476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.615597 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:21:15.615939 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.615666 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls podName:4167d24e-780d-478f-938b-6efe2af7bdb5 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:17.615649128 +0000 UTC m=+41.636732374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls") pod "cluster-samples-operator-65db858894-rfbpl" (UID: "4167d24e-780d-478f-938b-6efe2af7bdb5") : secret "samples-operator-tls" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616130 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616203 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:17.616183451 +0000 UTC m=+41.637266712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : configmap references non-existent config key: service-ca.crt Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616226 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls podName:1329b344-4e47-4d7a-b4c5-00fd4a60435e nodeName:}" failed. No retries permitted until 2026-07-09 14:21:17.616216524 +0000 UTC m=+41.637299772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4zmfm" (UID: "1329b344-4e47-4d7a-b4c5-00fd4a60435e") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616251 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616291 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert podName:25e5eacf-4b28-4bdb-aa9e-214257dfe58a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:17.6162746 +0000 UTC m=+41.637357844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wwjv6" (UID: "25e5eacf-4b28-4bdb-aa9e-214257dfe58a") : secret "networking-console-plugin-cert" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616293 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616324 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls podName:91ad122e-b32c-483b-89c3-fac94cbfc115 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:17.616314562 +0000 UTC m=+41.637397809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls") pod "dns-default-2fwcm" (UID: "91ad122e-b32c-483b-89c3-fac94cbfc115") : secret "dns-default-metrics-tls" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616386 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:21:15.616431 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:15.616418 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:17.616407946 +0000 UTC m=+41.637491190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : secret "router-metrics-certs-default" not found Jul 09 14:21:16.797154 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:16.796895 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" event={"ID":"7761ed1a-569d-497c-8f38-af325db1a0cf","Type":"ContainerStarted","Data":"636a9e1d07c7e81b296f83dd2cfd69ee7cb0e9e255d08c74c8262ab9796ff0dc"} Jul 09 14:21:16.815557 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:16.815451 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gm4nr" podStartSLOduration=23.935361167 podStartE2EDuration="25.815431578s" totalImagesPullingTime="1.880070411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:16.814935068 +0000 UTC m=+40.836018357" watchObservedRunningTime="2026-07-09 14:21:16.815431578 +0000 UTC m=+40.836514845" Jul 09 14:21:17.536738 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:17.536705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:17.536924 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.536873 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:21:17.536924 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.536893 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74d4c9b7c8-5tm8d: secret "image-registry-tls" not found Jul 09 14:21:17.537037 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.536958 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls podName:4e2a0f12-5b5a-47b2-8539-208a59706e7c nodeName:}" failed. No retries permitted until 2026-07-09 14:21:21.536938107 +0000 UTC m=+45.558021367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls") pod "image-registry-74d4c9b7c8-5tm8d" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c") : secret "image-registry-tls" not found Jul 09 14:21:17.637156 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:17.637127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:17.637299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:17.637184 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:17.637299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:17.637221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:17.637299 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637247 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:21:17.637299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:17.637255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637308 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls podName:91ad122e-b32c-483b-89c3-fac94cbfc115 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:21.637287461 +0000 UTC m=+45.658370711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls") pod "dns-default-2fwcm" (UID: "91ad122e-b32c-483b-89c3-fac94cbfc115") : secret "dns-default-metrics-tls" not found Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:17.637328 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637338 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637364 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637372 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:21.637361677 +0000 UTC m=+45.658444923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : secret "router-metrics-certs-default" not found Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637408 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637439 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:21.637430419 +0000 UTC m=+45.658513665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : configmap references non-existent config key: service-ca.crt Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637454 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert podName:25e5eacf-4b28-4bdb-aa9e-214257dfe58a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:21.637449149 +0000 UTC m=+45.658532391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wwjv6" (UID: "25e5eacf-4b28-4bdb-aa9e-214257dfe58a") : secret "networking-console-plugin-cert" not found Jul 09 14:21:17.637468 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637465 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls podName:4167d24e-780d-478f-938b-6efe2af7bdb5 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:21.637460139 +0000 UTC m=+45.658543381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls") pod "cluster-samples-operator-65db858894-rfbpl" (UID: "4167d24e-780d-478f-938b-6efe2af7bdb5") : secret "samples-operator-tls" not found Jul 09 14:21:17.637876 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:17.637526 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:17.637876 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637648 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:17.637876 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:17.637687 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls podName:1329b344-4e47-4d7a-b4c5-00fd4a60435e nodeName:}" failed. No retries permitted until 2026-07-09 14:21:21.637672887 +0000 UTC m=+45.658756130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4zmfm" (UID: "1329b344-4e47-4d7a-b4c5-00fd4a60435e") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:21.570214 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.570170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:21.570886 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.570290 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:21:21.570886 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.570304 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74d4c9b7c8-5tm8d: secret "image-registry-tls" not found Jul 09 14:21:21.570886 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.570361 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls podName:4e2a0f12-5b5a-47b2-8539-208a59706e7c nodeName:}" failed. No retries permitted until 2026-07-09 14:21:29.570344227 +0000 UTC m=+53.591427480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls") pod "image-registry-74d4c9b7c8-5tm8d" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c") : secret "image-registry-tls" not found Jul 09 14:21:21.671283 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.671259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:21.671389 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.671310 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:21.671389 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.671345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:21.671389 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.671372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:21.671525 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671420 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:29.6713998 +0000 UTC m=+53.692483057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : configmap references non-existent config key: service-ca.crt Jul 09 14:21:21.671525 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671485 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:21:21.671525 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.671507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:21.671668 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671533 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:29.671518073 +0000 UTC m=+53.692601329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : secret "router-metrics-certs-default" not found Jul 09 14:21:21.671668 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671485 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:21:21.671668 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.671593 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:21.671668 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671651 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert podName:25e5eacf-4b28-4bdb-aa9e-214257dfe58a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:29.671633696 +0000 UTC m=+53.692716944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wwjv6" (UID: "25e5eacf-4b28-4bdb-aa9e-214257dfe58a") : secret "networking-console-plugin-cert" not found Jul 09 14:21:21.671668 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671660 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:21:21.671856 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671698 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls podName:4167d24e-780d-478f-938b-6efe2af7bdb5 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:29.671686139 +0000 UTC m=+53.692769386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls") pod "cluster-samples-operator-65db858894-rfbpl" (UID: "4167d24e-780d-478f-938b-6efe2af7bdb5") : secret "samples-operator-tls" not found Jul 09 14:21:21.671856 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671701 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:21:21.671856 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671581 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:21.671856 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671738 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls podName:91ad122e-b32c-483b-89c3-fac94cbfc115 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:29.671727057 +0000 UTC m=+53.692810302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls") pod "dns-default-2fwcm" (UID: "91ad122e-b32c-483b-89c3-fac94cbfc115") : secret "dns-default-metrics-tls" not found Jul 09 14:21:21.671856 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:21.671765 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls podName:1329b344-4e47-4d7a-b4c5-00fd4a60435e nodeName:}" failed. No retries permitted until 2026-07-09 14:21:29.67174593 +0000 UTC m=+53.692829182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4zmfm" (UID: "1329b344-4e47-4d7a-b4c5-00fd4a60435e") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:21.815170 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.814808 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-bbxxs" event={"ID":"6966a58b-4631-44d0-8087-1bf9f97bdab3","Type":"ContainerStarted","Data":"c891cd8469be92e33f7232e2cd1296a6226009d61cff2d0f632b044284c2a50a"} Jul 09 14:21:21.817740 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.817713 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" event={"ID":"7ae68662-8dc0-48d2-ad61-b6e631f5fc81","Type":"ContainerStarted","Data":"b9dd94494801b8fc04154ca3c300cc5991c4a2524f834eb15a54158c7250159c"} Jul 09 14:21:21.839468 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:21.839357 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" podStartSLOduration=23.559643603 podStartE2EDuration="30.839338695s" totalImagesPullingTime="7.279695092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:21.838325085 +0000 UTC m=+45.859408350" watchObservedRunningTime="2026-07-09 14:21:21.839338695 +0000 UTC m=+45.860421952" Jul 09 14:21:22.829911 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.829869 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" event={"ID":"e5586c00-c5ed-458c-8001-abf514dda47b","Type":"ContainerStarted","Data":"b4652d8480fd8e5913ad0268d7ab1b1ef32fa85c32d520f63e287e1e2315a926"} Jul 09 14:21:22.833441 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.833405 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" event={"ID":"12ecb533-eadf-4be8-86f8-b8a843f0fd7e","Type":"ContainerStarted","Data":"3edf11fb7e39721f0cbdceb09c0cab387064872e3afbbeb3c396ca67308d3a50"} Jul 09 14:21:22.835263 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.835210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-27qjw" event={"ID":"9889133b-8a20-4733-a2ed-ea4f39c13ece","Type":"ContainerStarted","Data":"f9fd2e89d21fd50e9013f50aefb80cab1921bfa4cee41b41066a0724d8f0800e"} Jul 09 14:21:22.838139 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.837874 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/0.log" Jul 09 14:21:22.838139 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.837908 2563 generic.go:356] "Generic (PLEG): container finished" podID="6966a58b-4631-44d0-8087-1bf9f97bdab3" containerID="c891cd8469be92e33f7232e2cd1296a6226009d61cff2d0f632b044284c2a50a" exitCode=255 Jul 09 14:21:22.838139 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.837968 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-bbxxs" event={"ID":"6966a58b-4631-44d0-8087-1bf9f97bdab3","Type":"ContainerDied","Data":"c891cd8469be92e33f7232e2cd1296a6226009d61cff2d0f632b044284c2a50a"} Jul 09 14:21:22.838430 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.838414 2563 scope.go:117] "RemoveContainer" containerID="c891cd8469be92e33f7232e2cd1296a6226009d61cff2d0f632b044284c2a50a" Jul 09 14:21:22.840639 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.840595 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" event={"ID":"1858e3a2-d403-45fc-88d3-d18aca4057c6","Type":"ContainerStarted","Data":"f2bb45df00ea83d848e36ecda28435b67ee478658a857a57b166c399406a6bb8"} Jul 09 14:21:22.858130 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.856919 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jqssn"] Jul 09 14:21:22.858959 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.858918 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-m5nnq" podStartSLOduration=24.523016704 podStartE2EDuration="31.858901172s" totalImagesPullingTime="7.335884468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:22.85398119 +0000 UTC m=+46.875064477" watchObservedRunningTime="2026-07-09 14:21:22.858901172 +0000 UTC m=+46.879984438" Jul 09 14:21:22.890232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.889960 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jqssn"] Jul 09 14:21:22.890232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.890083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:22.893169 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.893144 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 14:21:22.893593 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.893569 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lsg92\"" Jul 09 14:21:22.893973 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.893733 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 14:21:22.893973 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.893613 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-27qjw" podStartSLOduration=2.6654581029999997 podStartE2EDuration="9.893595627s" totalImagesPullingTime="7.228137524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:22.892873673 +0000 UTC m=+46.913956931" watchObservedRunningTime="2026-07-09 14:21:22.893595627 +0000 UTC m=+46.914678893" Jul 09 14:21:22.917582 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.917542 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" podStartSLOduration=24.628619177 podStartE2EDuration="31.917529618s" totalImagesPullingTime="7.288910441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:22.916335468 +0000 UTC m=+46.937418731" watchObservedRunningTime="2026-07-09 14:21:22.917529618 +0000 UTC m=+46.938612882" Jul 09 14:21:22.936097 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.935183 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" podStartSLOduration=24.64249241 podStartE2EDuration="31.935158187s" totalImagesPullingTime="7.292665777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:22.933992475 +0000 UTC m=+46.955075742" watchObservedRunningTime="2026-07-09 14:21:22.935158187 +0000 UTC m=+46.956241452" Jul 09 14:21:22.985030 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.984992 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:22.985177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.985038 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51cf2858-a3fa-429c-b87c-ebb15481fdb1-data-volume\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:22.985177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.985074 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:22.985177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.985148 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51cf2858-a3fa-429c-b87c-ebb15481fdb1-crio-socket\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:22.985327 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:22.985238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rscb\" (UniqueName: \"kubernetes.io/projected/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-api-access-6rscb\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.085924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.085898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.086041 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.085939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51cf2858-a3fa-429c-b87c-ebb15481fdb1-data-volume\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.086041 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.085968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.086041 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.086002 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:21:23.086041 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.086035 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51cf2858-a3fa-429c-b87c-ebb15481fdb1-crio-socket\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.086228 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.086058 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:23.586041609 +0000 UTC m=+47.607124852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : secret "insights-runtime-extractor-tls" not found Jul 09 14:21:23.086228 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.086116 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:21:23.086228 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.086157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rscb\" (UniqueName: \"kubernetes.io/projected/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-api-access-6rscb\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.086228 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.086174 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:23.586157864 +0000 UTC m=+47.607241118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : configmap "kube-rbac-proxy" not found Jul 09 14:21:23.086397 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.086230 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51cf2858-a3fa-429c-b87c-ebb15481fdb1-crio-socket\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.086397 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.086297 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51cf2858-a3fa-429c-b87c-ebb15481fdb1-data-volume\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.096577 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.096461 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rscb\" (UniqueName: \"kubernetes.io/projected/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-api-access-6rscb\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.253773 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.253709 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj"] Jul 09 14:21:23.272770 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.272745 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj"] Jul 09 14:21:23.272887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.272857 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" Jul 09 14:21:23.275390 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.275359 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 14:21:23.275390 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.275366 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 14:21:23.275547 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.275481 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qlf45\"" Jul 09 14:21:23.389036 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.389012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsg2h\" (UniqueName: \"kubernetes.io/projected/04cbc9f6-c3d3-4509-8449-cc0b191e212a-kube-api-access-rsg2h\") pod \"migrator-7f59b54f6c-55jxj\" (UID: \"04cbc9f6-c3d3-4509-8449-cc0b191e212a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" Jul 09 14:21:23.490486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.490120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsg2h\" (UniqueName: \"kubernetes.io/projected/04cbc9f6-c3d3-4509-8449-cc0b191e212a-kube-api-access-rsg2h\") pod \"migrator-7f59b54f6c-55jxj\" (UID: \"04cbc9f6-c3d3-4509-8449-cc0b191e212a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" Jul 09 14:21:23.499814 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.499791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsg2h\" (UniqueName: \"kubernetes.io/projected/04cbc9f6-c3d3-4509-8449-cc0b191e212a-kube-api-access-rsg2h\") pod \"migrator-7f59b54f6c-55jxj\" (UID: \"04cbc9f6-c3d3-4509-8449-cc0b191e212a\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" Jul 09 14:21:23.582431 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.582401 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" Jul 09 14:21:23.590819 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.590799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.590917 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.590833 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:23.590917 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.590895 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:21:23.591030 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.590943 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:24.590931025 +0000 UTC m=+48.612014268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : configmap "kube-rbac-proxy" not found Jul 09 14:21:23.591030 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.590956 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:21:23.591030 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.591015 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:24.590997986 +0000 UTC m=+48.612081242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : secret "insights-runtime-extractor-tls" not found Jul 09 14:21:23.721796 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.721575 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj"] Jul 09 14:21:23.725797 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:23.725769 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cbc9f6_c3d3_4509_8449_cc0b191e212a.slice/crio-15dd33e068ab446e650ef5faf7e0a6c765f1fdf38df0f8ced3ca0bbedc13f073 WatchSource:0}: Error finding container 15dd33e068ab446e650ef5faf7e0a6c765f1fdf38df0f8ced3ca0bbedc13f073: Status 404 returned error can't find the container with id 15dd33e068ab446e650ef5faf7e0a6c765f1fdf38df0f8ced3ca0bbedc13f073 Jul 09 14:21:23.844257 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.844180 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" event={"ID":"04cbc9f6-c3d3-4509-8449-cc0b191e212a","Type":"ContainerStarted","Data":"15dd33e068ab446e650ef5faf7e0a6c765f1fdf38df0f8ced3ca0bbedc13f073"} Jul 09 14:21:23.845796 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.845780 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/1.log" Jul 09 14:21:23.846163 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.846148 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/0.log" Jul 09 14:21:23.846236 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.846183 2563 generic.go:356] "Generic (PLEG): container finished" podID="6966a58b-4631-44d0-8087-1bf9f97bdab3" containerID="4d2539e3a01f39cad80db341c6c5a9a1499c231634ce87cf2d144d7f0835bf89" exitCode=255 Jul 09 14:21:23.846289 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.846270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-bbxxs" event={"ID":"6966a58b-4631-44d0-8087-1bf9f97bdab3","Type":"ContainerDied","Data":"4d2539e3a01f39cad80db341c6c5a9a1499c231634ce87cf2d144d7f0835bf89"} Jul 09 14:21:23.846327 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.846309 2563 scope.go:117] "RemoveContainer" containerID="c891cd8469be92e33f7232e2cd1296a6226009d61cff2d0f632b044284c2a50a" Jul 09 14:21:23.846470 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:23.846452 2563 scope.go:117] "RemoveContainer" containerID="4d2539e3a01f39cad80db341c6c5a9a1499c231634ce87cf2d144d7f0835bf89" Jul 09 14:21:23.846699 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:23.846656 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-bbxxs_openshift-console-operator(6966a58b-4631-44d0-8087-1bf9f97bdab3)\"" pod="openshift-console-operator/console-operator-7866497494-bbxxs" podUID="6966a58b-4631-44d0-8087-1bf9f97bdab3" Jul 09 14:21:24.202813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.202741 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:24.202813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.202811 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:24.274753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.274729 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sx47j_d6a5d9db-5636-48cd-bb96-76b36e182c4b/dns-node-resolver/0.log" Jul 09 14:21:24.599592 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.599563 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:24.599749 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.599597 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:24.599749 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:24.599694 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:21:24.599749 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:24.599728 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:21:24.599749 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:24.599744 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:26.599731387 +0000 UTC m=+50.620814629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : configmap "kube-rbac-proxy" not found Jul 09 14:21:24.599959 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:24.599786 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:26.599769072 +0000 UTC m=+50.620852324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : secret "insights-runtime-extractor-tls" not found Jul 09 14:21:24.850703 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.850646 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/1.log" Jul 09 14:21:24.851022 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.850949 2563 scope.go:117] "RemoveContainer" containerID="4d2539e3a01f39cad80db341c6c5a9a1499c231634ce87cf2d144d7f0835bf89" Jul 09 14:21:24.851099 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:24.851080 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-bbxxs_openshift-console-operator(6966a58b-4631-44d0-8087-1bf9f97bdab3)\"" pod="openshift-console-operator/console-operator-7866497494-bbxxs" podUID="6966a58b-4631-44d0-8087-1bf9f97bdab3" Jul 09 14:21:24.875095 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:24.875079 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b4jwm_7ca81f21-58d3-478f-a38e-afd97d8aff04/node-ca/0.log" Jul 09 14:21:25.674559 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:25.674534 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-27qjw_9889133b-8a20-4733-a2ed-ea4f39c13ece/serve-healthcheck-canary/0.log" Jul 09 14:21:25.856330 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:25.856288 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" event={"ID":"04cbc9f6-c3d3-4509-8449-cc0b191e212a","Type":"ContainerStarted","Data":"bcdb8f73f62e9d4e52bf2729e5f923f9d90a30eed9c5a372b131fc3b2d8d8e48"} Jul 09 14:21:25.856703 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:25.856611 2563 scope.go:117] "RemoveContainer" containerID="4d2539e3a01f39cad80db341c6c5a9a1499c231634ce87cf2d144d7f0835bf89" Jul 09 14:21:25.856843 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:25.856809 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-bbxxs_openshift-console-operator(6966a58b-4631-44d0-8087-1bf9f97bdab3)\"" pod="openshift-console-operator/console-operator-7866497494-bbxxs" podUID="6966a58b-4631-44d0-8087-1bf9f97bdab3" Jul 09 14:21:26.618680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:26.618653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:26.618808 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:26.618687 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:26.618867 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:26.618810 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:21:26.618867 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:26.618825 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:21:26.618867 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:26.618862 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:30.618849522 +0000 UTC m=+54.639932765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : configmap "kube-rbac-proxy" not found Jul 09 14:21:26.619025 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:26.618872 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:30.618867237 +0000 UTC m=+54.639950479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : secret "insights-runtime-extractor-tls" not found Jul 09 14:21:26.860445 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:26.860419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" event={"ID":"04cbc9f6-c3d3-4509-8449-cc0b191e212a","Type":"ContainerStarted","Data":"fd39f70037bd7b7e8233750c9fcf9de28397c7ab27e8b89d757b2beba0d27528"} Jul 09 14:21:29.644052 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:29.644016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:29.644509 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.644172 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 14:21:29.644509 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.644190 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74d4c9b7c8-5tm8d: secret "image-registry-tls" not found Jul 09 14:21:29.644509 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.644245 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls podName:4e2a0f12-5b5a-47b2-8539-208a59706e7c nodeName:}" failed. No retries permitted until 2026-07-09 14:21:45.644229244 +0000 UTC m=+69.665312488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls") pod "image-registry-74d4c9b7c8-5tm8d" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c") : secret "image-registry-tls" not found Jul 09 14:21:29.744381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:29.744345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:29.744503 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:29.744407 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:29.744503 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:29.744446 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:29.744503 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744471 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:29.744503 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:29.744482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:29.744515 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744525 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls podName:1329b344-4e47-4d7a-b4c5-00fd4a60435e nodeName:}" failed. No retries permitted until 2026-07-09 14:21:45.744511309 +0000 UTC m=+69.765594567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-4zmfm" (UID: "1329b344-4e47-4d7a-b4c5-00fd4a60435e") : secret "cluster-monitoring-operator-tls" not found Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:29.744555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744580 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744604 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:45.744582747 +0000 UTC m=+69.765665993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : configmap references non-existent config key: service-ca.crt Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744655 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert podName:25e5eacf-4b28-4bdb-aa9e-214257dfe58a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:45.744639909 +0000 UTC m=+69.765723152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-wwjv6" (UID: "25e5eacf-4b28-4bdb-aa9e-214257dfe58a") : secret "networking-console-plugin-cert" not found Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744680 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744706 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744730 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs podName:5a553e49-b8e4-4543-9550-9eba5165cc3a nodeName:}" failed. No retries permitted until 2026-07-09 14:21:45.74471515 +0000 UTC m=+69.765798393 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs") pod "router-default-5bdf54bb-m5z9v" (UID: "5a553e49-b8e4-4543-9550-9eba5165cc3a") : secret "router-metrics-certs-default" not found Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744727 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 14:21:29.744753 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744752 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls podName:91ad122e-b32c-483b-89c3-fac94cbfc115 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:45.744740705 +0000 UTC m=+69.765823953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls") pod "dns-default-2fwcm" (UID: "91ad122e-b32c-483b-89c3-fac94cbfc115") : secret "dns-default-metrics-tls" not found Jul 09 14:21:29.745201 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:29.744779 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls podName:4167d24e-780d-478f-938b-6efe2af7bdb5 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:45.744767363 +0000 UTC m=+69.765850609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls") pod "cluster-samples-operator-65db858894-rfbpl" (UID: "4167d24e-780d-478f-938b-6efe2af7bdb5") : secret "samples-operator-tls" not found Jul 09 14:21:30.651214 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:30.651180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:30.651568 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:30.651229 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:30.651568 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:30.651335 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:21:30.651568 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:30.651372 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 14:21:30.651568 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:30.651398 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:38.65138197 +0000 UTC m=+62.672465215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : secret "insights-runtime-extractor-tls" not found Jul 09 14:21:30.651568 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:30.651425 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:38.651411626 +0000 UTC m=+62.672494871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : configmap "kube-rbac-proxy" not found Jul 09 14:21:38.717155 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:38.717114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:38.717155 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:38.717163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:38.717810 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:38.717289 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 14:21:38.717810 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:38.717391 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls podName:51cf2858-a3fa-429c-b87c-ebb15481fdb1 nodeName:}" failed. No retries permitted until 2026-07-09 14:21:54.71736821 +0000 UTC m=+78.738451506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqssn" (UID: "51cf2858-a3fa-429c-b87c-ebb15481fdb1") : secret "insights-runtime-extractor-tls" not found Jul 09 14:21:38.717810 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:38.717678 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51cf2858-a3fa-429c-b87c-ebb15481fdb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:39.569277 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.569253 2563 scope.go:117] "RemoveContainer" containerID="4d2539e3a01f39cad80db341c6c5a9a1499c231634ce87cf2d144d7f0835bf89" Jul 09 14:21:39.768757 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.768734 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rzbg4" Jul 09 14:21:39.799920 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.799877 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-55jxj" podStartSLOduration=14.792565696 podStartE2EDuration="16.799863877s" totalImagesPullingTime="2.007298181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:26.880486613 +0000 UTC m=+50.901569876" watchObservedRunningTime="2026-07-09 14:21:39.799863877 +0000 UTC m=+63.820947140" Jul 09 14:21:39.905169 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.905118 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:21:39.905560 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.905544 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/1.log" Jul 09 14:21:39.905607 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.905577 2563 generic.go:356] "Generic (PLEG): container finished" podID="6966a58b-4631-44d0-8087-1bf9f97bdab3" containerID="8c83cebdb7b596271f0468b987c7e72c095929b7d0aea10de38dbf78b18ee76f" exitCode=255 Jul 09 14:21:39.905680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.905661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-bbxxs" event={"ID":"6966a58b-4631-44d0-8087-1bf9f97bdab3","Type":"ContainerDied","Data":"8c83cebdb7b596271f0468b987c7e72c095929b7d0aea10de38dbf78b18ee76f"} Jul 09 14:21:39.905734 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.905714 2563 scope.go:117] "RemoveContainer" containerID="4d2539e3a01f39cad80db341c6c5a9a1499c231634ce87cf2d144d7f0835bf89" Jul 09 14:21:39.905999 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:39.905973 2563 scope.go:117] "RemoveContainer" containerID="8c83cebdb7b596271f0468b987c7e72c095929b7d0aea10de38dbf78b18ee76f" Jul 09 14:21:39.906164 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:39.906143 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-bbxxs_openshift-console-operator(6966a58b-4631-44d0-8087-1bf9f97bdab3)\"" pod="openshift-console-operator/console-operator-7866497494-bbxxs" podUID="6966a58b-4631-44d0-8087-1bf9f97bdab3" Jul 09 14:21:40.911024 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:40.911000 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:21:42.245404 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.245376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:42.245789 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.245532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:42.248043 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.248021 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 14:21:42.248888 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.248871 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 14:21:42.258467 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.258447 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e527178e-badd-4625-a67a-f5e2904fbc82-original-pull-secret\") pod \"global-pull-secret-syncer-928hb\" (UID: \"e527178e-badd-4625-a67a-f5e2904fbc82\") " pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:42.258467 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.258460 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4154366d-8c15-403c-8df9-cceea3424319-metrics-certs\") pod \"network-metrics-daemon-fb2rh\" (UID: \"4154366d-8c15-403c-8df9-cceea3424319\") " pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:42.288848 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.288825 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4gsp\"" Jul 09 14:21:42.297672 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.297654 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fb2rh" Jul 09 14:21:42.297792 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.297777 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-928hb" Jul 09 14:21:42.347534 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.347429 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:42.354784 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.354240 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdnw\" (UniqueName: \"kubernetes.io/projected/7a682c4e-eed0-4463-b5d3-b91351ebbe96-kube-api-access-6vdnw\") pod \"network-check-target-95fkk\" (UID: \"7a682c4e-eed0-4463-b5d3-b91351ebbe96\") " pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:42.441183 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.441133 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-928hb"] Jul 09 14:21:42.443142 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:42.443116 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode527178e_badd_4625_a67a_f5e2904fbc82.slice/crio-f0315bfb1e5fb666d337ee2134a1885de49f14b1ddad74fd9654942608ec74fd WatchSource:0}: Error finding container f0315bfb1e5fb666d337ee2134a1885de49f14b1ddad74fd9654942608ec74fd: Status 404 returned error can't find the container with id f0315bfb1e5fb666d337ee2134a1885de49f14b1ddad74fd9654942608ec74fd Jul 09 14:21:42.456016 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.455992 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fb2rh"] Jul 09 14:21:42.458730 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:42.458700 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4154366d_8c15_403c_8df9_cceea3424319.slice/crio-3caf9c31656dc993c9d7fe35a0966b84cf1501364990a4fe2d5ada7aad6c3731 WatchSource:0}: Error finding container 3caf9c31656dc993c9d7fe35a0966b84cf1501364990a4fe2d5ada7aad6c3731: Status 404 returned error can't find the container with id 3caf9c31656dc993c9d7fe35a0966b84cf1501364990a4fe2d5ada7aad6c3731 Jul 09 14:21:42.608283 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.608259 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bcf26\"" Jul 09 14:21:42.616313 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.616291 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:42.733000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.732977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-95fkk"] Jul 09 14:21:42.735693 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:42.735667 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a682c4e_eed0_4463_b5d3_b91351ebbe96.slice/crio-49826ab9549a82660e018f9ba3c331013426068528d52bcc634e9ae0637cba97 WatchSource:0}: Error finding container 49826ab9549a82660e018f9ba3c331013426068528d52bcc634e9ae0637cba97: Status 404 returned error can't find the container with id 49826ab9549a82660e018f9ba3c331013426068528d52bcc634e9ae0637cba97 Jul 09 14:21:42.918362 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.918296 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95fkk" event={"ID":"7a682c4e-eed0-4463-b5d3-b91351ebbe96","Type":"ContainerStarted","Data":"d784d7cb324bfcef1e0fad89ac1f9fc09c49fef113e34690ffb06a48306ed45b"} Jul 09 14:21:42.918362 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.918353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-95fkk" event={"ID":"7a682c4e-eed0-4463-b5d3-b91351ebbe96","Type":"ContainerStarted","Data":"49826ab9549a82660e018f9ba3c331013426068528d52bcc634e9ae0637cba97"} Jul 09 14:21:42.918511 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.918378 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:21:42.919526 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.919505 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fb2rh" event={"ID":"4154366d-8c15-403c-8df9-cceea3424319","Type":"ContainerStarted","Data":"3caf9c31656dc993c9d7fe35a0966b84cf1501364990a4fe2d5ada7aad6c3731"} Jul 09 14:21:42.920578 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.920560 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-928hb" event={"ID":"e527178e-badd-4625-a67a-f5e2904fbc82","Type":"ContainerStarted","Data":"f0315bfb1e5fb666d337ee2134a1885de49f14b1ddad74fd9654942608ec74fd"} Jul 09 14:21:42.934841 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:42.934798 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-95fkk" podStartSLOduration=66.934784088 podStartE2EDuration="1m6.934784088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:42.934597616 +0000 UTC m=+66.955680881" watchObservedRunningTime="2026-07-09 14:21:42.934784088 +0000 UTC m=+66.955867352" Jul 09 14:21:44.203495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:44.203456 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:44.203937 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:44.203515 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:21:44.203937 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:44.203851 2563 scope.go:117] "RemoveContainer" containerID="8c83cebdb7b596271f0468b987c7e72c095929b7d0aea10de38dbf78b18ee76f" Jul 09 14:21:44.204060 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:44.204033 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-bbxxs_openshift-console-operator(6966a58b-4631-44d0-8087-1bf9f97bdab3)\"" pod="openshift-console-operator/console-operator-7866497494-bbxxs" podUID="6966a58b-4631-44d0-8087-1bf9f97bdab3" Jul 09 14:21:44.929708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:44.929671 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fb2rh" event={"ID":"4154366d-8c15-403c-8df9-cceea3424319","Type":"ContainerStarted","Data":"b0e84957d3f1d6b10f52474440301b1d442a0247cebf41b48cc82e1200fb3615"} Jul 09 14:21:44.929842 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:44.929739 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fb2rh" event={"ID":"4154366d-8c15-403c-8df9-cceea3424319","Type":"ContainerStarted","Data":"37552475de3a90965e7a23335dbdade9c1b3bb31607fbb0cb6317ebdc4e394be"} Jul 09 14:21:44.946133 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:44.946090 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fb2rh" podStartSLOduration=66.864648858 podStartE2EDuration="1m8.946075304s" totalImagesPullingTime="2.081426446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:44.944926093 +0000 UTC m=+68.966009369" watchObservedRunningTime="2026-07-09 14:21:44.946075304 +0000 UTC m=+68.967158567" Jul 09 14:21:45.677398 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.677365 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:45.679993 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.679966 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"image-registry-74d4c9b7c8-5tm8d\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:45.778460 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.778434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:45.778590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.778475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:45.778590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.778502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:45.778590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.778537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:45.778590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.778565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:45.778590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.778585 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:45.779219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.779182 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a553e49-b8e4-4543-9550-9eba5165cc3a-service-ca-bundle\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:45.781188 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.781140 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4167d24e-780d-478f-938b-6efe2af7bdb5-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rfbpl\" (UID: \"4167d24e-780d-478f-938b-6efe2af7bdb5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:45.781302 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.781240 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/91ad122e-b32c-483b-89c3-fac94cbfc115-metrics-tls\") pod \"dns-default-2fwcm\" (UID: \"91ad122e-b32c-483b-89c3-fac94cbfc115\") " pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:45.781386 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.781362 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1329b344-4e47-4d7a-b4c5-00fd4a60435e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-4zmfm\" (UID: \"1329b344-4e47-4d7a-b4c5-00fd4a60435e\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:45.781471 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.781365 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25e5eacf-4b28-4bdb-aa9e-214257dfe58a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wwjv6\" (UID: \"25e5eacf-4b28-4bdb-aa9e-214257dfe58a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:45.781726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.781702 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a553e49-b8e4-4543-9550-9eba5165cc3a-metrics-certs\") pod \"router-default-5bdf54bb-m5z9v\" (UID: \"5a553e49-b8e4-4543-9550-9eba5165cc3a\") " pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:45.899724 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.899702 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-khbqc\"" Jul 09 14:21:45.907819 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.907797 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:45.962987 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.962961 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x5fqd\"" Jul 09 14:21:45.968592 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.968563 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kv6gx\"" Jul 09 14:21:45.971306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.971103 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" Jul 09 14:21:45.975443 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.975395 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxvxp\"" Jul 09 14:21:45.976428 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.976406 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" Jul 09 14:21:45.983346 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.983323 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:45.985785 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.985755 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x8gpr\"" Jul 09 14:21:45.995037 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:45.994819 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:46.006327 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:46.006121 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w8kmw\"" Jul 09 14:21:46.014450 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:46.014026 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" Jul 09 14:21:46.052485 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:46.052132 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74d4c9b7c8-5tm8d"] Jul 09 14:21:46.104704 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:46.104673 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm"] Jul 09 14:21:47.326003 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:47.325969 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1329b344_4e47_4d7a_b4c5_00fd4a60435e.slice/crio-9575d3f455cfb7771388ec45aa8d76b2a4429d471d2a75ccd8e6e3a36e2594eb WatchSource:0}: Error finding container 9575d3f455cfb7771388ec45aa8d76b2a4429d471d2a75ccd8e6e3a36e2594eb: Status 404 returned error can't find the container with id 9575d3f455cfb7771388ec45aa8d76b2a4429d471d2a75ccd8e6e3a36e2594eb Jul 09 14:21:47.326695 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:47.326670 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2a0f12_5b5a_47b2_8539_208a59706e7c.slice/crio-7419e83cf83756de569af51aa4e60f62688a41fbd67b71b0d8df68722843a4ea WatchSource:0}: Error finding container 7419e83cf83756de569af51aa4e60f62688a41fbd67b71b0d8df68722843a4ea: Status 404 returned error can't find the container with id 7419e83cf83756de569af51aa4e60f62688a41fbd67b71b0d8df68722843a4ea Jul 09 14:21:47.522707 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.522607 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl"] Jul 09 14:21:47.719413 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.719380 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fwcm"] Jul 09 14:21:47.721928 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:47.721898 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ad122e_b32c_483b_89c3_fac94cbfc115.slice/crio-5d11bb0c481889b66b0f227f68a2cdd106d3f2dba04e5c05a9070e446062ed8e WatchSource:0}: Error finding container 5d11bb0c481889b66b0f227f68a2cdd106d3f2dba04e5c05a9070e446062ed8e: Status 404 returned error can't find the container with id 5d11bb0c481889b66b0f227f68a2cdd106d3f2dba04e5c05a9070e446062ed8e Jul 09 14:21:47.763697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.763661 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wwjv6"] Jul 09 14:21:47.766287 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:47.766256 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e5eacf_4b28_4bdb_aa9e_214257dfe58a.slice/crio-b1714958288402c438b644f26710290a52362460aaade558592db493375fcf66 WatchSource:0}: Error finding container b1714958288402c438b644f26710290a52362460aaade558592db493375fcf66: Status 404 returned error can't find the container with id b1714958288402c438b644f26710290a52362460aaade558592db493375fcf66 Jul 09 14:21:47.775649 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.775599 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5bdf54bb-m5z9v"] Jul 09 14:21:47.778427 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:47.778403 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a553e49_b8e4_4543_9550_9eba5165cc3a.slice/crio-db55f051b3a4e2c66ebbb54d2fde1ef8773002d56bf01cec15fd24c84bc43ab7 WatchSource:0}: Error finding container db55f051b3a4e2c66ebbb54d2fde1ef8773002d56bf01cec15fd24c84bc43ab7: Status 404 returned error can't find the container with id db55f051b3a4e2c66ebbb54d2fde1ef8773002d56bf01cec15fd24c84bc43ab7 Jul 09 14:21:47.941195 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.941162 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" event={"ID":"5a553e49-b8e4-4543-9550-9eba5165cc3a","Type":"ContainerStarted","Data":"88b5522a2395960707e3a85cd788c993f7be2133ae11924ac36caeb9f41f0cf7"} Jul 09 14:21:47.941352 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.941208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" event={"ID":"5a553e49-b8e4-4543-9550-9eba5165cc3a","Type":"ContainerStarted","Data":"db55f051b3a4e2c66ebbb54d2fde1ef8773002d56bf01cec15fd24c84bc43ab7"} Jul 09 14:21:47.942370 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.942341 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" event={"ID":"1329b344-4e47-4d7a-b4c5-00fd4a60435e","Type":"ContainerStarted","Data":"9575d3f455cfb7771388ec45aa8d76b2a4429d471d2a75ccd8e6e3a36e2594eb"} Jul 09 14:21:47.943352 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.943329 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fwcm" event={"ID":"91ad122e-b32c-483b-89c3-fac94cbfc115","Type":"ContainerStarted","Data":"5d11bb0c481889b66b0f227f68a2cdd106d3f2dba04e5c05a9070e446062ed8e"} Jul 09 14:21:47.944379 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.944359 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" event={"ID":"25e5eacf-4b28-4bdb-aa9e-214257dfe58a","Type":"ContainerStarted","Data":"b1714958288402c438b644f26710290a52362460aaade558592db493375fcf66"} Jul 09 14:21:47.945399 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.945378 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" event={"ID":"4167d24e-780d-478f-938b-6efe2af7bdb5","Type":"ContainerStarted","Data":"132a739de2576e66a8bb7289192886082d99c21c3032cfac048b80a9205bb080"} Jul 09 14:21:47.946752 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.946734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" event={"ID":"4e2a0f12-5b5a-47b2-8539-208a59706e7c","Type":"ContainerStarted","Data":"860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1"} Jul 09 14:21:47.946847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.946764 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" event={"ID":"4e2a0f12-5b5a-47b2-8539-208a59706e7c","Type":"ContainerStarted","Data":"7419e83cf83756de569af51aa4e60f62688a41fbd67b71b0d8df68722843a4ea"} Jul 09 14:21:47.946847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.946814 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:21:47.948014 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.947995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-928hb" event={"ID":"e527178e-badd-4625-a67a-f5e2904fbc82","Type":"ContainerStarted","Data":"027e649276b6310a61dbfaa651d5722fde6c71e71d2756822a299b9909eb354c"} Jul 09 14:21:47.971100 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.971029 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" podStartSLOduration=56.971015313 podStartE2EDuration="56.971015313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:47.970027908 +0000 UTC m=+71.991111192" watchObservedRunningTime="2026-07-09 14:21:47.971015313 +0000 UTC m=+71.992098578" Jul 09 14:21:47.986853 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.986812 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-928hb" podStartSLOduration=67.056209857 podStartE2EDuration="1m11.986800519s" totalImagesPullingTime="4.930590662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:47.986543319 +0000 UTC m=+72.007626600" watchObservedRunningTime="2026-07-09 14:21:47.986800519 +0000 UTC m=+72.007883774" Jul 09 14:21:47.996219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.996198 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:47.998438 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:47.998422 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:48.008737 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:48.008700 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" podStartSLOduration=71.008688876 podStartE2EDuration="1m11.008688876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:48.007758956 +0000 UTC m=+72.028842245" watchObservedRunningTime="2026-07-09 14:21:48.008688876 +0000 UTC m=+72.029772161" Jul 09 14:21:48.953105 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:48.953039 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:48.954233 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:48.954200 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5bdf54bb-m5z9v" Jul 09 14:21:49.959827 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:49.959788 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" event={"ID":"25e5eacf-4b28-4bdb-aa9e-214257dfe58a","Type":"ContainerStarted","Data":"32bf2408ff604ab74e26e150882b2f3b51ba2a29c87060c9fe10aab80af286d1"} Jul 09 14:21:49.963956 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:49.963921 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" event={"ID":"1329b344-4e47-4d7a-b4c5-00fd4a60435e","Type":"ContainerStarted","Data":"dad5d5218183c1550dced19eeb2459a54300748bec745a466fc9894f6345fa9e"} Jul 09 14:21:49.976088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:49.976040 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-wwjv6" podStartSLOduration=54.254935384 podStartE2EDuration="55.976020496s" totalImagesPullingTime="1.721085112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:49.975200414 +0000 UTC m=+73.996283673" watchObservedRunningTime="2026-07-09 14:21:49.976020496 +0000 UTC m=+73.997103762" Jul 09 14:21:49.992958 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:49.992914 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-4zmfm" podStartSLOduration=56.828540145 podStartE2EDuration="58.992899893s" totalImagesPullingTime="2.164359748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:49.992210485 +0000 UTC m=+74.013293749" watchObservedRunningTime="2026-07-09 14:21:49.992899893 +0000 UTC m=+74.013983156" Jul 09 14:21:50.970307 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:50.970265 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" event={"ID":"4167d24e-780d-478f-938b-6efe2af7bdb5","Type":"ContainerStarted","Data":"34a63246f4b93b4d25a01f8d4ae5f62508a4347a39bcdbeeb3951c1b00940e03"} Jul 09 14:21:50.970748 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:50.970325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" event={"ID":"4167d24e-780d-478f-938b-6efe2af7bdb5","Type":"ContainerStarted","Data":"dc83dcf267e3a271d2dcfadccc32acf1beadf17d306eaf77bbc36930e01978dc"} Jul 09 14:21:50.988945 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:50.988899 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rfbpl" podStartSLOduration=57.377666032 podStartE2EDuration="59.988880842s" totalImagesPullingTime="2.61121481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:50.98851817 +0000 UTC m=+75.009601448" watchObservedRunningTime="2026-07-09 14:21:50.988880842 +0000 UTC m=+75.009964107" Jul 09 14:21:51.975016 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:51.974983 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fwcm" event={"ID":"91ad122e-b32c-483b-89c3-fac94cbfc115","Type":"ContainerStarted","Data":"48bb892c57b577065b2aa61942c674d01e63fc0fa2de03de59ada1fd6b14fba9"} Jul 09 14:21:51.975375 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:51.975056 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fwcm" event={"ID":"91ad122e-b32c-483b-89c3-fac94cbfc115","Type":"ContainerStarted","Data":"4e5a9ee6a41da50448ff6ddec9eb147239ba4223a21d69a84c01cf466ea022cb"} Jul 09 14:21:51.975375 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:51.975264 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2fwcm" Jul 09 14:21:54.573317 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:54.573288 2563 scope.go:117] "RemoveContainer" containerID="8c83cebdb7b596271f0468b987c7e72c095929b7d0aea10de38dbf78b18ee76f" Jul 09 14:21:54.573710 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:21:54.573432 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-bbxxs_openshift-console-operator(6966a58b-4631-44d0-8087-1bf9f97bdab3)\"" pod="openshift-console-operator/console-operator-7866497494-bbxxs" podUID="6966a58b-4631-44d0-8087-1bf9f97bdab3" Jul 09 14:21:54.753060 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:54.753030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:54.755663 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:54.755613 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51cf2858-a3fa-429c-b87c-ebb15481fdb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqssn\" (UID: \"51cf2858-a3fa-429c-b87c-ebb15481fdb1\") " pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:55.011645 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:55.011603 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lsg92\"" Jul 09 14:21:55.018966 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:55.018946 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jqssn" Jul 09 14:21:55.169238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:55.169192 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2fwcm" podStartSLOduration=38.699889156 podStartE2EDuration="42.16917519s" totalImagesPullingTime="3.469286034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:21:51.998063916 +0000 UTC m=+76.019147208" watchObservedRunningTime="2026-07-09 14:21:55.16917519 +0000 UTC m=+79.190258453" Jul 09 14:21:55.170208 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:55.170189 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jqssn"] Jul 09 14:21:55.172230 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:21:55.172204 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cf2858_a3fa_429c_b87c_ebb15481fdb1.slice/crio-90a45c0e2f8c5209082e033fe78932d40f16bc48b114223811d56be46b3e5dcf WatchSource:0}: Error finding container 90a45c0e2f8c5209082e033fe78932d40f16bc48b114223811d56be46b3e5dcf: Status 404 returned error can't find the container with id 90a45c0e2f8c5209082e033fe78932d40f16bc48b114223811d56be46b3e5dcf Jul 09 14:21:55.992728 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:21:55.992702 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqssn" event={"ID":"51cf2858-a3fa-429c-b87c-ebb15481fdb1","Type":"ContainerStarted","Data":"90a45c0e2f8c5209082e033fe78932d40f16bc48b114223811d56be46b3e5dcf"} Jul 09 14:22:00.010820 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:00.010784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqssn" event={"ID":"51cf2858-a3fa-429c-b87c-ebb15481fdb1","Type":"ContainerStarted","Data":"a41c8653d71aea608a6a9b17fdee29b282d9fa52cc310a7e14e99be554307954"} Jul 09 14:22:00.011168 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:00.010844 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqssn" event={"ID":"51cf2858-a3fa-429c-b87c-ebb15481fdb1","Type":"ContainerStarted","Data":"8f2202cf3420af6dd389a9db8c808a9d7259ebc88c4c4ce37eb572325e346ece"} Jul 09 14:22:01.455082 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.455041 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vdv22"] Jul 09 14:22:01.458410 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.458389 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.460927 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.460896 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 14:22:01.461082 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.461051 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5nhnb\"" Jul 09 14:22:01.461144 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.461089 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 14:22:01.461977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.461954 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 14:22:01.462083 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.461955 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 14:22:01.502524 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502484 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502655 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-sys\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502655 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502573 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-textfile\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502762 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502762 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-root\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502762 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502718 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-tls\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502920 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502770 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4r5\" (UniqueName: \"kubernetes.io/projected/8a06c082-ca75-4ad7-aa94-6800b0c1d051-kube-api-access-dh4r5\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502920 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-wtmp\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.502920 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.502852 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a06c082-ca75-4ad7-aa94-6800b0c1d051-metrics-client-ca\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603103 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-tls\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603110 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4r5\" (UniqueName: \"kubernetes.io/projected/8a06c082-ca75-4ad7-aa94-6800b0c1d051-kube-api-access-dh4r5\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-wtmp\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a06c082-ca75-4ad7-aa94-6800b0c1d051-metrics-client-ca\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-sys\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603227 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-textfile\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603290 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-wtmp\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603298 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-root\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603416 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-root\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.603533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603460 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a06c082-ca75-4ad7-aa94-6800b0c1d051-sys\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.604049 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603793 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-textfile\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.604049 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603840 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a06c082-ca75-4ad7-aa94-6800b0c1d051-metrics-client-ca\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.604049 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.603995 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.605557 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.605538 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-tls\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.605680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.605595 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a06c082-ca75-4ad7-aa94-6800b0c1d051-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.613855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.613829 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4r5\" (UniqueName: \"kubernetes.io/projected/8a06c082-ca75-4ad7-aa94-6800b0c1d051-kube-api-access-dh4r5\") pod \"node-exporter-vdv22\" (UID: \"8a06c082-ca75-4ad7-aa94-6800b0c1d051\") " pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:01.768878 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:01.768839 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vdv22" Jul 09 14:22:02.291280 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:22:02.291251 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a06c082_ca75_4ad7_aa94_6800b0c1d051.slice/crio-b2cfed7f008f3627846d0280022a5c35f4f0ff3a8bbf509c15e7f24d1bff04cb WatchSource:0}: Error finding container b2cfed7f008f3627846d0280022a5c35f4f0ff3a8bbf509c15e7f24d1bff04cb: Status 404 returned error can't find the container with id b2cfed7f008f3627846d0280022a5c35f4f0ff3a8bbf509c15e7f24d1bff04cb Jul 09 14:22:02.985915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:02.985884 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2fwcm" Jul 09 14:22:03.024761 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:03.024727 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdv22" event={"ID":"8a06c082-ca75-4ad7-aa94-6800b0c1d051","Type":"ContainerStarted","Data":"b2cfed7f008f3627846d0280022a5c35f4f0ff3a8bbf509c15e7f24d1bff04cb"} Jul 09 14:22:03.028061 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:03.027490 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqssn" event={"ID":"51cf2858-a3fa-429c-b87c-ebb15481fdb1","Type":"ContainerStarted","Data":"ff62cadce620dc65ffc709e929e975317ce735836fdf83d19fa8f06aeb25e687"} Jul 09 14:22:03.045699 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:03.045651 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jqssn" podStartSLOduration=34.005860044 podStartE2EDuration="41.045631295s" totalImagesPullingTime="7.039771251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 14:21:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:03.045218941 +0000 UTC m=+87.066302205" watchObservedRunningTime="2026-07-09 14:22:03.045631295 +0000 UTC m=+87.066714551" Jul 09 14:22:04.033540 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:04.033508 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a06c082-ca75-4ad7-aa94-6800b0c1d051" containerID="3f85d62ce6b394efc3f00f664c194e89b5ac2a6effa4dd19ba9d257b23956b46" exitCode=0 Jul 09 14:22:04.033932 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:04.033608 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdv22" event={"ID":"8a06c082-ca75-4ad7-aa94-6800b0c1d051","Type":"ContainerDied","Data":"3f85d62ce6b394efc3f00f664c194e89b5ac2a6effa4dd19ba9d257b23956b46"} Jul 09 14:22:05.044528 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:05.044489 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdv22" event={"ID":"8a06c082-ca75-4ad7-aa94-6800b0c1d051","Type":"ContainerStarted","Data":"6fd7a2dd0e513ebe5b93c1d65b5e2ed45bc1aed825d465e2fbfa8009b1faedde"} Jul 09 14:22:05.044937 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:05.044555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdv22" event={"ID":"8a06c082-ca75-4ad7-aa94-6800b0c1d051","Type":"ContainerStarted","Data":"c26852b5243d8a37fa9ca3e0649ddb15d1aa21138adb3c7e8e4346613c91bc3f"} Jul 09 14:22:05.065522 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:05.065469 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vdv22" podStartSLOduration=3.101055532 podStartE2EDuration="4.065450705s" totalImagesPullingTime="903.64559ms" totalInitContainerRuntime="60.749583ms" isStatefulPod=true podCreationTimestamp="2026-07-09 14:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:05.064369262 +0000 UTC m=+89.085452549" watchObservedRunningTime="2026-07-09 14:22:05.065450705 +0000 UTC m=+89.086533971" Jul 09 14:22:06.570828 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:06.570801 2563 scope.go:117] "RemoveContainer" containerID="8c83cebdb7b596271f0468b987c7e72c095929b7d0aea10de38dbf78b18ee76f" Jul 09 14:22:07.054325 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.054302 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:22:07.054449 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.054375 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-bbxxs" event={"ID":"6966a58b-4631-44d0-8087-1bf9f97bdab3","Type":"ContainerStarted","Data":"0a1f728f65d4adadb8f50c768b51305b0a0e3105f1746ebb0534c9a72b549a5f"} Jul 09 14:22:07.054670 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.054648 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:22:07.077025 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.076983 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-bbxxs" podStartSLOduration=68.858866073 podStartE2EDuration="1m16.076969266s" totalImagesPullingTime="7.218103193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:07.075771788 +0000 UTC m=+91.096855073" watchObservedRunningTime="2026-07-09 14:22:07.076969266 +0000 UTC m=+91.098052530" Jul 09 14:22:07.725119 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.725096 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-bbxxs" Jul 09 14:22:07.898707 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.898674 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-ljw9j"] Jul 09 14:22:07.900867 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.900847 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-ljw9j" Jul 09 14:22:07.903197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.903178 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gbw2j\"" Jul 09 14:22:07.903464 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.903443 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 14:22:07.903517 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.903455 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 14:22:07.910950 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.910933 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-ljw9j"] Jul 09 14:22:07.948455 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:07.948429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfxd\" (UniqueName: \"kubernetes.io/projected/8711347f-783c-4d25-92ed-b1ef6095c77a-kube-api-access-bjfxd\") pod \"downloads-68c9cbb497-ljw9j\" (UID: \"8711347f-783c-4d25-92ed-b1ef6095c77a\") " pod="openshift-console/downloads-68c9cbb497-ljw9j" Jul 09 14:22:08.049312 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:08.049283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfxd\" (UniqueName: \"kubernetes.io/projected/8711347f-783c-4d25-92ed-b1ef6095c77a-kube-api-access-bjfxd\") pod \"downloads-68c9cbb497-ljw9j\" (UID: \"8711347f-783c-4d25-92ed-b1ef6095c77a\") " pod="openshift-console/downloads-68c9cbb497-ljw9j" Jul 09 14:22:08.057799 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:08.057781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfxd\" (UniqueName: \"kubernetes.io/projected/8711347f-783c-4d25-92ed-b1ef6095c77a-kube-api-access-bjfxd\") pod \"downloads-68c9cbb497-ljw9j\" (UID: \"8711347f-783c-4d25-92ed-b1ef6095c77a\") " pod="openshift-console/downloads-68c9cbb497-ljw9j" Jul 09 14:22:08.209652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:08.209633 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-ljw9j" Jul 09 14:22:08.331576 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:08.331547 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-ljw9j"] Jul 09 14:22:08.333967 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:22:08.333927 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8711347f_783c_4d25_92ed_b1ef6095c77a.slice/crio-0b5ad21def71452498cb931ea157f67ced6b9b7428893ba73429a69828bb1905 WatchSource:0}: Error finding container 0b5ad21def71452498cb931ea157f67ced6b9b7428893ba73429a69828bb1905: Status 404 returned error can't find the container with id 0b5ad21def71452498cb931ea157f67ced6b9b7428893ba73429a69828bb1905 Jul 09 14:22:08.956909 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:08.956883 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:22:09.063660 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:09.063612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-ljw9j" event={"ID":"8711347f-783c-4d25-92ed-b1ef6095c77a","Type":"ContainerStarted","Data":"0b5ad21def71452498cb931ea157f67ced6b9b7428893ba73429a69828bb1905"} Jul 09 14:22:09.432527 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:09.432496 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74d4c9b7c8-5tm8d"] Jul 09 14:22:13.927000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:13.926969 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-95fkk" Jul 09 14:22:18.431401 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.431364 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-696f678954-tqrcp"] Jul 09 14:22:18.450946 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.450918 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696f678954-tqrcp"] Jul 09 14:22:18.451112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.451007 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.453706 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.453680 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 14:22:18.453706 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.453700 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 14:22:18.453882 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.453700 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rgbhl\"" Jul 09 14:22:18.454500 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.454473 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 14:22:18.454500 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.454486 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 14:22:18.454773 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.454501 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 14:22:18.531638 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.531598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-serving-cert\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.531805 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.531669 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-service-ca\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.531805 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.531725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-oauth-config\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.531805 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.531774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-config\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.531957 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.531814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-oauth-serving-cert\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.531957 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.531848 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpv6\" (UniqueName: \"kubernetes.io/projected/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-kube-api-access-vlpv6\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.632842 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.632813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-serving-cert\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.632977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.632871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-service-ca\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.632977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.632911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-oauth-config\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.632977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.632944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-config\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.633134 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.632982 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-oauth-serving-cert\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.633134 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.633033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpv6\" (UniqueName: \"kubernetes.io/projected/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-kube-api-access-vlpv6\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.633728 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.633700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-service-ca\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.633833 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.633776 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-config\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.633833 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.633802 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-oauth-serving-cert\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.635513 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.635492 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-oauth-config\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.635600 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.635549 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-serving-cert\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.642310 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.642287 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpv6\" (UniqueName: \"kubernetes.io/projected/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-kube-api-access-vlpv6\") pod \"console-696f678954-tqrcp\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:18.762217 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:18.762183 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:25.289937 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:25.289912 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696f678954-tqrcp"] Jul 09 14:22:25.292220 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:22:25.292192 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd47ad10_7b3e_49d4_bb0e_cd8974e1987e.slice/crio-56fe727c95f88a5ef2436ac0fb0b9e8e10a4f7263645f6c13a3171cffed89dea WatchSource:0}: Error finding container 56fe727c95f88a5ef2436ac0fb0b9e8e10a4f7263645f6c13a3171cffed89dea: Status 404 returned error can't find the container with id 56fe727c95f88a5ef2436ac0fb0b9e8e10a4f7263645f6c13a3171cffed89dea Jul 09 14:22:26.143117 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:26.142969 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-ljw9j" event={"ID":"8711347f-783c-4d25-92ed-b1ef6095c77a","Type":"ContainerStarted","Data":"475aceba04c386dfd35b8ad397742962324bc4a8d2958d0918f3b3af52956673"} Jul 09 14:22:26.143513 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:26.143471 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-ljw9j" Jul 09 14:22:26.146500 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:26.146472 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696f678954-tqrcp" event={"ID":"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e","Type":"ContainerStarted","Data":"56fe727c95f88a5ef2436ac0fb0b9e8e10a4f7263645f6c13a3171cffed89dea"} Jul 09 14:22:26.156248 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:26.156222 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-ljw9j" Jul 09 14:22:26.180427 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:26.180165 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-ljw9j" podStartSLOduration=2.267127026 podStartE2EDuration="19.180145422s" totalImagesPullingTime="16.913018396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:26.162205296 +0000 UTC m=+110.183288583" watchObservedRunningTime="2026-07-09 14:22:26.180145422 +0000 UTC m=+110.201228687" Jul 09 14:22:27.017304 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.015541 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-867dccd8fd-h8lgj"] Jul 09 14:22:27.042248 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.042152 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867dccd8fd-h8lgj"] Jul 09 14:22:27.042248 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.042247 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.052804 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.052464 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 14:22:27.107696 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.107638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-oauth-serving-cert\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.107896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.107705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-serving-cert\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.107896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.107798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mn7z\" (UniqueName: \"kubernetes.io/projected/fa5980cb-604f-458f-b8ed-d93f6816755e-kube-api-access-6mn7z\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.107896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.107849 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-trusted-ca-bundle\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.107896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.107885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-service-ca\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.108112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.107927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-console-config\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.108112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.107955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-oauth-config\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.208568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-console-config\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.208650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-oauth-config\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.208728 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-oauth-serving-cert\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.208758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-serving-cert\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.208803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mn7z\" (UniqueName: \"kubernetes.io/projected/fa5980cb-604f-458f-b8ed-d93f6816755e-kube-api-access-6mn7z\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.208837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-trusted-ca-bundle\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.208868 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-service-ca\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.210954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.209977 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-console-config\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.212167 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.212138 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-oauth-serving-cert\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.212340 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.212307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-trusted-ca-bundle\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.212772 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.212744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-service-ca\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.213048 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.213005 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-oauth-config\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.221200 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.217881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-serving-cert\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.221200 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.220384 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mn7z\" (UniqueName: \"kubernetes.io/projected/fa5980cb-604f-458f-b8ed-d93f6816755e-kube-api-access-6mn7z\") pod \"console-867dccd8fd-h8lgj\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:27.356392 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:27.356297 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:28.762120 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:28.762094 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867dccd8fd-h8lgj"] Jul 09 14:22:28.764590 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:22:28.764556 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5980cb_604f_458f_b8ed_d93f6816755e.slice/crio-dc5d458319b38dc53651a019d676e4712126a8bc9702162bf2e6b5bd254581cf WatchSource:0}: Error finding container dc5d458319b38dc53651a019d676e4712126a8bc9702162bf2e6b5bd254581cf: Status 404 returned error can't find the container with id dc5d458319b38dc53651a019d676e4712126a8bc9702162bf2e6b5bd254581cf Jul 09 14:22:29.166501 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:29.166420 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867dccd8fd-h8lgj" event={"ID":"fa5980cb-604f-458f-b8ed-d93f6816755e","Type":"ContainerStarted","Data":"dc5d458319b38dc53651a019d676e4712126a8bc9702162bf2e6b5bd254581cf"} Jul 09 14:22:29.168289 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:29.168260 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696f678954-tqrcp" event={"ID":"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e","Type":"ContainerStarted","Data":"cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce"} Jul 09 14:22:29.193891 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:29.193844 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-696f678954-tqrcp" podStartSLOduration=7.567526189 podStartE2EDuration="11.193830127s" totalImagesPullingTime="3.626303938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:29.192309838 +0000 UTC m=+113.213393127" watchObservedRunningTime="2026-07-09 14:22:29.193830127 +0000 UTC m=+113.214913391" Jul 09 14:22:30.175320 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:30.175277 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867dccd8fd-h8lgj" event={"ID":"fa5980cb-604f-458f-b8ed-d93f6816755e","Type":"ContainerStarted","Data":"eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e"} Jul 09 14:22:30.195138 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:30.195089 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-867dccd8fd-h8lgj" podStartSLOduration=3.723730217 podStartE2EDuration="4.195068498s" totalImagesPullingTime="471.338281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:22:30.194739427 +0000 UTC m=+114.215822691" watchObservedRunningTime="2026-07-09 14:22:30.195068498 +0000 UTC m=+114.216151763" Jul 09 14:22:34.455367 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.455320 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" podUID="4e2a0f12-5b5a-47b2-8539-208a59706e7c" containerName="registry" containerID="cri-o://860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1" gracePeriod=30 Jul 09 14:22:34.744760 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.744721 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:22:34.878222 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-certificates\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878222 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878229 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br488\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-kube-api-access-br488\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878256 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878295 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-installation-pull-secrets\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878318 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-image-registry-private-configuration\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878343 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-trusted-ca\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878379 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e2a0f12-5b5a-47b2-8539-208a59706e7c-ca-trust-extracted\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878406 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-bound-sa-token\") pod \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\" (UID: \"4e2a0f12-5b5a-47b2-8539-208a59706e7c\") " Jul 09 14:22:34.878894 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878738 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:22:34.878894 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.878810 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:22:34.880857 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.880828 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:22:34.881010 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.880923 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:22:34.881010 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.880995 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:22:34.881086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.881033 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:22:34.881249 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.881232 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-kube-api-access-br488" (OuterVolumeSpecName: "kube-api-access-br488") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "kube-api-access-br488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:22:34.889546 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.889506 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2a0f12-5b5a-47b2-8539-208a59706e7c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4e2a0f12-5b5a-47b2-8539-208a59706e7c" (UID: "4e2a0f12-5b5a-47b2-8539-208a59706e7c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979710 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-bound-sa-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979740 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-certificates\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979750 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br488\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-kube-api-access-br488\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979758 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e2a0f12-5b5a-47b2-8539-208a59706e7c-registry-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979766 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-installation-pull-secrets\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979776 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e2a0f12-5b5a-47b2-8539-208a59706e7c-image-registry-private-configuration\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979784 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a0f12-5b5a-47b2-8539-208a59706e7c-trusted-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:34.979790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:34.979792 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e2a0f12-5b5a-47b2-8539-208a59706e7c-ca-trust-extracted\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:22:35.196459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.196419 2563 generic.go:356] "Generic (PLEG): container finished" podID="4e2a0f12-5b5a-47b2-8539-208a59706e7c" containerID="860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1" exitCode=0 Jul 09 14:22:35.196633 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.196469 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" Jul 09 14:22:35.196633 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.196507 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" event={"ID":"4e2a0f12-5b5a-47b2-8539-208a59706e7c","Type":"ContainerDied","Data":"860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1"} Jul 09 14:22:35.196633 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.196558 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74d4c9b7c8-5tm8d" event={"ID":"4e2a0f12-5b5a-47b2-8539-208a59706e7c","Type":"ContainerDied","Data":"7419e83cf83756de569af51aa4e60f62688a41fbd67b71b0d8df68722843a4ea"} Jul 09 14:22:35.196633 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.196582 2563 scope.go:117] "RemoveContainer" containerID="860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1" Jul 09 14:22:35.205276 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.205258 2563 scope.go:117] "RemoveContainer" containerID="860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1" Jul 09 14:22:35.205547 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:22:35.205520 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1\": container with ID starting with 860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1 not found: ID does not exist" containerID="860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1" Jul 09 14:22:35.205627 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.205560 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1"} err="failed to get container status \"860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1\": rpc error: code = NotFound desc = could not find container \"860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1\": container with ID starting with 860b37366890bb231b06e08e9140043d48c3108c365cba04c438993f744741b1 not found: ID does not exist" Jul 09 14:22:35.217275 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.217253 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74d4c9b7c8-5tm8d"] Jul 09 14:22:35.220914 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:35.220896 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74d4c9b7c8-5tm8d"] Jul 09 14:22:36.573968 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:36.573938 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2a0f12-5b5a-47b2-8539-208a59706e7c" path="/var/lib/kubelet/pods/4e2a0f12-5b5a-47b2-8539-208a59706e7c/volumes" Jul 09 14:22:37.357393 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:37.357355 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:37.357578 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:37.357424 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:37.362764 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:37.362233 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:38.218567 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:38.218538 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:22:38.272019 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:38.271989 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696f678954-tqrcp"] Jul 09 14:22:38.763102 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:38.763069 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:22:43.234836 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:43.234753 2563 generic.go:356] "Generic (PLEG): container finished" podID="1858e3a2-d403-45fc-88d3-d18aca4057c6" containerID="f2bb45df00ea83d848e36ecda28435b67ee478658a857a57b166c399406a6bb8" exitCode=0 Jul 09 14:22:43.235241 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:43.234834 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" event={"ID":"1858e3a2-d403-45fc-88d3-d18aca4057c6","Type":"ContainerDied","Data":"f2bb45df00ea83d848e36ecda28435b67ee478658a857a57b166c399406a6bb8"} Jul 09 14:22:43.235241 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:43.235174 2563 scope.go:117] "RemoveContainer" containerID="f2bb45df00ea83d848e36ecda28435b67ee478658a857a57b166c399406a6bb8" Jul 09 14:22:43.236535 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:43.236515 2563 generic.go:356] "Generic (PLEG): container finished" podID="12ecb533-eadf-4be8-86f8-b8a843f0fd7e" containerID="3edf11fb7e39721f0cbdceb09c0cab387064872e3afbbeb3c396ca67308d3a50" exitCode=0 Jul 09 14:22:43.236643 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:43.236549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" event={"ID":"12ecb533-eadf-4be8-86f8-b8a843f0fd7e","Type":"ContainerDied","Data":"3edf11fb7e39721f0cbdceb09c0cab387064872e3afbbeb3c396ca67308d3a50"} Jul 09 14:22:43.236788 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:43.236776 2563 scope.go:117] "RemoveContainer" containerID="3edf11fb7e39721f0cbdceb09c0cab387064872e3afbbeb3c396ca67308d3a50" Jul 09 14:22:44.241510 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:44.241477 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2fgcw" event={"ID":"12ecb533-eadf-4be8-86f8-b8a843f0fd7e","Type":"ContainerStarted","Data":"e77421a5c0fb255c3fadb7e090e14141f86edd357c88a7836b300a0dfd17ddba"} Jul 09 14:22:44.243193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:44.243169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-88h2b" event={"ID":"1858e3a2-d403-45fc-88d3-d18aca4057c6","Type":"ContainerStarted","Data":"e1f5185cc2218d2579c82e9488a542122cff03b778525c8917c9af772defdbcd"} Jul 09 14:22:48.259660 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:48.259601 2563 generic.go:356] "Generic (PLEG): container finished" podID="7ae68662-8dc0-48d2-ad61-b6e631f5fc81" containerID="b9dd94494801b8fc04154ca3c300cc5991c4a2524f834eb15a54158c7250159c" exitCode=0 Jul 09 14:22:48.260140 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:48.259671 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" event={"ID":"7ae68662-8dc0-48d2-ad61-b6e631f5fc81","Type":"ContainerDied","Data":"b9dd94494801b8fc04154ca3c300cc5991c4a2524f834eb15a54158c7250159c"} Jul 09 14:22:48.260140 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:48.259987 2563 scope.go:117] "RemoveContainer" containerID="b9dd94494801b8fc04154ca3c300cc5991c4a2524f834eb15a54158c7250159c" Jul 09 14:22:49.265099 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:22:49.265059 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-fwnfg" event={"ID":"7ae68662-8dc0-48d2-ad61-b6e631f5fc81","Type":"ContainerStarted","Data":"eaea33adea15c2b9c8cbe3ab481559c5bcf46ef9aa91d6f4deb9ab2ac835a5c5"} Jul 09 14:23:03.293013 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.292948 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-696f678954-tqrcp" podUID="dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" containerName="console" containerID="cri-o://cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce" gracePeriod=15 Jul 09 14:23:03.574412 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.574348 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:23:03.613401 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613369 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-serving-cert\") pod \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " Jul 09 14:23:03.613543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613409 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-config\") pod \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " Jul 09 14:23:03.613543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613441 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-oauth-config\") pod \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " Jul 09 14:23:03.613543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613459 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-oauth-serving-cert\") pod \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " Jul 09 14:23:03.613543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613492 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-service-ca\") pod \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " Jul 09 14:23:03.613543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613511 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlpv6\" (UniqueName: \"kubernetes.io/projected/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-kube-api-access-vlpv6\") pod \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\" (UID: \"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e\") " Jul 09 14:23:03.613865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613839 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-config" (OuterVolumeSpecName: "console-config") pod "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" (UID: "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:23:03.613943 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613918 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" (UID: "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:23:03.613996 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.613954 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-service-ca" (OuterVolumeSpecName: "service-ca") pod "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" (UID: "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:23:03.615581 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.615556 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" (UID: "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:23:03.615704 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.615686 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-kube-api-access-vlpv6" (OuterVolumeSpecName: "kube-api-access-vlpv6") pod "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" (UID: "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e"). InnerVolumeSpecName "kube-api-access-vlpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:23:03.615751 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.615742 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" (UID: "dd47ad10-7b3e-49d4-bb0e-cd8974e1987e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:23:03.714160 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.714128 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:23:03.714160 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.714154 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:23:03.714160 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.714163 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-console-oauth-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:23:03.714343 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.714172 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-oauth-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:23:03.714343 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.714182 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:23:03.714343 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:03.714189 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpv6\" (UniqueName: \"kubernetes.io/projected/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e-kube-api-access-vlpv6\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:23:04.323709 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.323678 2563 generic.go:356] "Generic (PLEG): container finished" podID="dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" containerID="cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce" exitCode=0 Jul 09 14:23:04.324123 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.323727 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696f678954-tqrcp" Jul 09 14:23:04.324123 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.323740 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696f678954-tqrcp" event={"ID":"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e","Type":"ContainerDied","Data":"cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce"} Jul 09 14:23:04.324123 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.323784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696f678954-tqrcp" event={"ID":"dd47ad10-7b3e-49d4-bb0e-cd8974e1987e","Type":"ContainerDied","Data":"56fe727c95f88a5ef2436ac0fb0b9e8e10a4f7263645f6c13a3171cffed89dea"} Jul 09 14:23:04.324123 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.323804 2563 scope.go:117] "RemoveContainer" containerID="cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce" Jul 09 14:23:04.340508 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.340487 2563 scope.go:117] "RemoveContainer" containerID="cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce" Jul 09 14:23:04.340801 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:23:04.340783 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce\": container with ID starting with cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce not found: ID does not exist" containerID="cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce" Jul 09 14:23:04.340864 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.340810 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce"} err="failed to get container status \"cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce\": rpc error: code = NotFound desc = could not find container \"cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce\": container with ID starting with cb8c96256adb14546d473f3012dc10774951e9d6d7f57222abb5b80ca55d97ce not found: ID does not exist" Jul 09 14:23:04.343797 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.343777 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696f678954-tqrcp"] Jul 09 14:23:04.347845 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.347820 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-696f678954-tqrcp"] Jul 09 14:23:04.578083 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:04.574895 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" path="/var/lib/kubelet/pods/dd47ad10-7b3e-49d4-bb0e-cd8974e1987e/volumes" Jul 09 14:23:30.638354 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.638213 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bdfdcc99b-6p8js"] Jul 09 14:23:30.638896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.638535 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" containerName="console" Jul 09 14:23:30.638896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.638547 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" containerName="console" Jul 09 14:23:30.638896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.638555 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2a0f12-5b5a-47b2-8539-208a59706e7c" containerName="registry" Jul 09 14:23:30.638896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.638560 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2a0f12-5b5a-47b2-8539-208a59706e7c" containerName="registry" Jul 09 14:23:30.638896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.638609 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd47ad10-7b3e-49d4-bb0e-cd8974e1987e" containerName="console" Jul 09 14:23:30.638896 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.638633 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2a0f12-5b5a-47b2-8539-208a59706e7c" containerName="registry" Jul 09 14:23:30.644028 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.642868 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.648493 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.648467 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdfdcc99b-6p8js"] Jul 09 14:23:30.715168 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.715130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-trusted-ca-bundle\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.715168 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.715170 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-oauth-config\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.715367 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.715185 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68fw\" (UniqueName: \"kubernetes.io/projected/10cc1183-842f-4773-998c-ca7edf3490ed-kube-api-access-z68fw\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.715367 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.715231 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-service-ca\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.715367 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.715276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-console-config\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.715367 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.715327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-serving-cert\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.715485 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.715366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-oauth-serving-cert\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.816491 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.816440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-serving-cert\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.816491 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.816500 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-oauth-serving-cert\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.816753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.816533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-trusted-ca-bundle\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.816753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.816550 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-oauth-config\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.816753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.816565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68fw\" (UniqueName: \"kubernetes.io/projected/10cc1183-842f-4773-998c-ca7edf3490ed-kube-api-access-z68fw\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.816753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.816590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-service-ca\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.816753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.816613 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-console-config\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.817402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.817375 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-service-ca\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.817505 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.817427 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-console-config\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.817505 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.817462 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-oauth-serving-cert\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.817590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.817508 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-trusted-ca-bundle\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.819544 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.819519 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-oauth-config\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.819657 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.819525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-serving-cert\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.825001 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.824978 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68fw\" (UniqueName: \"kubernetes.io/projected/10cc1183-842f-4773-998c-ca7edf3490ed-kube-api-access-z68fw\") pod \"console-7bdfdcc99b-6p8js\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:30.954529 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:30.954429 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:31.081340 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:31.081306 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdfdcc99b-6p8js"] Jul 09 14:23:31.084125 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:23:31.084101 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cc1183_842f_4773_998c_ca7edf3490ed.slice/crio-57008db1b87ecdeb390fb5c255f7eb2bb8d2474862eb17736e4e1e7502852af6 WatchSource:0}: Error finding container 57008db1b87ecdeb390fb5c255f7eb2bb8d2474862eb17736e4e1e7502852af6: Status 404 returned error can't find the container with id 57008db1b87ecdeb390fb5c255f7eb2bb8d2474862eb17736e4e1e7502852af6 Jul 09 14:23:31.429908 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:31.429867 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdfdcc99b-6p8js" event={"ID":"10cc1183-842f-4773-998c-ca7edf3490ed","Type":"ContainerStarted","Data":"36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0"} Jul 09 14:23:31.430077 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:31.429932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdfdcc99b-6p8js" event={"ID":"10cc1183-842f-4773-998c-ca7edf3490ed","Type":"ContainerStarted","Data":"57008db1b87ecdeb390fb5c255f7eb2bb8d2474862eb17736e4e1e7502852af6"} Jul 09 14:23:31.449487 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:31.449434 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bdfdcc99b-6p8js" podStartSLOduration=1.449421216 podStartE2EDuration="1.449421216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:23:31.448378991 +0000 UTC m=+175.469462280" watchObservedRunningTime="2026-07-09 14:23:31.449421216 +0000 UTC m=+175.470504479" Jul 09 14:23:40.955001 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:40.954963 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:40.955483 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:40.955033 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:40.959733 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:40.959713 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:41.472020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:41.471991 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:23:41.528228 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:23:41.528199 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-867dccd8fd-h8lgj"] Jul 09 14:24:06.553258 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.553210 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-867dccd8fd-h8lgj" podUID="fa5980cb-604f-458f-b8ed-d93f6816755e" containerName="console" containerID="cri-o://eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e" gracePeriod=15 Jul 09 14:24:06.793990 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.793964 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:24:06.882054 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.881985 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mn7z\" (UniqueName: \"kubernetes.io/projected/fa5980cb-604f-458f-b8ed-d93f6816755e-kube-api-access-6mn7z\") pod \"fa5980cb-604f-458f-b8ed-d93f6816755e\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " Jul 09 14:24:06.882054 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882029 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-oauth-serving-cert\") pod \"fa5980cb-604f-458f-b8ed-d93f6816755e\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " Jul 09 14:24:06.882054 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882046 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-console-config\") pod \"fa5980cb-604f-458f-b8ed-d93f6816755e\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " Jul 09 14:24:06.882245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882081 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-oauth-config\") pod \"fa5980cb-604f-458f-b8ed-d93f6816755e\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " Jul 09 14:24:06.882245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882113 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-trusted-ca-bundle\") pod \"fa5980cb-604f-458f-b8ed-d93f6816755e\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " Jul 09 14:24:06.882245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882168 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-serving-cert\") pod \"fa5980cb-604f-458f-b8ed-d93f6816755e\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " Jul 09 14:24:06.882245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882188 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-service-ca\") pod \"fa5980cb-604f-458f-b8ed-d93f6816755e\" (UID: \"fa5980cb-604f-458f-b8ed-d93f6816755e\") " Jul 09 14:24:06.882543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882490 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fa5980cb-604f-458f-b8ed-d93f6816755e" (UID: "fa5980cb-604f-458f-b8ed-d93f6816755e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:24:06.882683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882600 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-service-ca" (OuterVolumeSpecName: "service-ca") pod "fa5980cb-604f-458f-b8ed-d93f6816755e" (UID: "fa5980cb-604f-458f-b8ed-d93f6816755e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:24:06.882683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882598 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-console-config" (OuterVolumeSpecName: "console-config") pod "fa5980cb-604f-458f-b8ed-d93f6816755e" (UID: "fa5980cb-604f-458f-b8ed-d93f6816755e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:24:06.882683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.882644 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fa5980cb-604f-458f-b8ed-d93f6816755e" (UID: "fa5980cb-604f-458f-b8ed-d93f6816755e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:24:06.884018 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.883992 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5980cb-604f-458f-b8ed-d93f6816755e-kube-api-access-6mn7z" (OuterVolumeSpecName: "kube-api-access-6mn7z") pod "fa5980cb-604f-458f-b8ed-d93f6816755e" (UID: "fa5980cb-604f-458f-b8ed-d93f6816755e"). InnerVolumeSpecName "kube-api-access-6mn7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:24:06.884117 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.884091 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fa5980cb-604f-458f-b8ed-d93f6816755e" (UID: "fa5980cb-604f-458f-b8ed-d93f6816755e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:24:06.884160 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.884104 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fa5980cb-604f-458f-b8ed-d93f6816755e" (UID: "fa5980cb-604f-458f-b8ed-d93f6816755e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:24:06.983297 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.983270 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mn7z\" (UniqueName: \"kubernetes.io/projected/fa5980cb-604f-458f-b8ed-d93f6816755e-kube-api-access-6mn7z\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:24:06.983297 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.983298 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-oauth-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:24:06.983414 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.983307 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-console-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:24:06.983414 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.983315 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-oauth-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:24:06.983414 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.983324 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-trusted-ca-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:24:06.983414 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.983332 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa5980cb-604f-458f-b8ed-d93f6816755e-console-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:24:06.983414 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:06.983340 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5980cb-604f-458f-b8ed-d93f6816755e-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:24:07.567797 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.567762 2563 generic.go:356] "Generic (PLEG): container finished" podID="fa5980cb-604f-458f-b8ed-d93f6816755e" containerID="eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e" exitCode=0 Jul 09 14:24:07.568219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.567812 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867dccd8fd-h8lgj" event={"ID":"fa5980cb-604f-458f-b8ed-d93f6816755e","Type":"ContainerDied","Data":"eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e"} Jul 09 14:24:07.568219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.567850 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867dccd8fd-h8lgj" event={"ID":"fa5980cb-604f-458f-b8ed-d93f6816755e","Type":"ContainerDied","Data":"dc5d458319b38dc53651a019d676e4712126a8bc9702162bf2e6b5bd254581cf"} Jul 09 14:24:07.568219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.567871 2563 scope.go:117] "RemoveContainer" containerID="eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e" Jul 09 14:24:07.568219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.567814 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867dccd8fd-h8lgj" Jul 09 14:24:07.576859 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.576837 2563 scope.go:117] "RemoveContainer" containerID="eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e" Jul 09 14:24:07.577125 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:24:07.577107 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e\": container with ID starting with eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e not found: ID does not exist" containerID="eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e" Jul 09 14:24:07.577192 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.577136 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e"} err="failed to get container status \"eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e\": rpc error: code = NotFound desc = could not find container \"eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e\": container with ID starting with eb3c0def0168c56721f8e476350f9a7f08a9699a324206ed19014eb6b713f05e not found: ID does not exist" Jul 09 14:24:07.587836 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.587806 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-867dccd8fd-h8lgj"] Jul 09 14:24:07.591630 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:07.591590 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-867dccd8fd-h8lgj"] Jul 09 14:24:08.575196 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:08.575148 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5980cb-604f-458f-b8ed-d93f6816755e" path="/var/lib/kubelet/pods/fa5980cb-604f-458f-b8ed-d93f6816755e/volumes" Jul 09 14:24:55.943135 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:55.943096 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77954cdf95-5b59p"] Jul 09 14:24:55.943750 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:55.943442 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5980cb-604f-458f-b8ed-d93f6816755e" containerName="console" Jul 09 14:24:55.943750 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:55.943460 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5980cb-604f-458f-b8ed-d93f6816755e" containerName="console" Jul 09 14:24:55.943750 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:55.943523 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5980cb-604f-458f-b8ed-d93f6816755e" containerName="console" Jul 09 14:24:55.946147 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:55.946121 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:55.968325 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:55.968298 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77954cdf95-5b59p"] Jul 09 14:24:56.034388 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.034364 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-trusted-ca-bundle\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.034493 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.034392 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-serving-cert\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.034493 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.034410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-config\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.034493 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.034424 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-oauth-serving-cert\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.034599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.034513 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-oauth-config\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.034599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.034542 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-service-ca\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.034599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.034567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgrs\" (UniqueName: \"kubernetes.io/projected/33ff4139-ffc9-45ca-9f48-9ab908365df3-kube-api-access-9zgrs\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.134946 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.134912 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-trusted-ca-bundle\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135066 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.134951 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-serving-cert\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135066 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.134966 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-config\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135066 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.134981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-oauth-serving-cert\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135218 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.135108 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-oauth-config\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135218 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.135152 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-service-ca\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135218 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.135187 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgrs\" (UniqueName: \"kubernetes.io/projected/33ff4139-ffc9-45ca-9f48-9ab908365df3-kube-api-access-9zgrs\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135719 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.135693 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-config\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135810 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.135735 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-oauth-serving-cert\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.135954 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.135936 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-service-ca\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.136014 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.135992 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-trusted-ca-bundle\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.137538 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.137516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-serving-cert\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.137641 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.137537 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-oauth-config\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.143272 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.143249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgrs\" (UniqueName: \"kubernetes.io/projected/33ff4139-ffc9-45ca-9f48-9ab908365df3-kube-api-access-9zgrs\") pod \"console-77954cdf95-5b59p\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.256937 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.256916 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:24:56.380846 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.380819 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77954cdf95-5b59p"] Jul 09 14:24:56.383836 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:24:56.383809 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ff4139_ffc9_45ca_9f48_9ab908365df3.slice/crio-ebc2a12a5a046cdcd39bbc5a1f2c27815e1950578aad3ddc67c942451142a1e6 WatchSource:0}: Error finding container ebc2a12a5a046cdcd39bbc5a1f2c27815e1950578aad3ddc67c942451142a1e6: Status 404 returned error can't find the container with id ebc2a12a5a046cdcd39bbc5a1f2c27815e1950578aad3ddc67c942451142a1e6 Jul 09 14:24:56.761353 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.761323 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77954cdf95-5b59p" event={"ID":"33ff4139-ffc9-45ca-9f48-9ab908365df3","Type":"ContainerStarted","Data":"0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75"} Jul 09 14:24:56.761508 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.761370 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77954cdf95-5b59p" event={"ID":"33ff4139-ffc9-45ca-9f48-9ab908365df3","Type":"ContainerStarted","Data":"ebc2a12a5a046cdcd39bbc5a1f2c27815e1950578aad3ddc67c942451142a1e6"} Jul 09 14:24:56.781161 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:24:56.781116 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77954cdf95-5b59p" podStartSLOduration=1.781100509 podStartE2EDuration="1.781100509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:24:56.779914777 +0000 UTC m=+260.800998053" watchObservedRunningTime="2026-07-09 14:24:56.781100509 +0000 UTC m=+260.802183772" Jul 09 14:25:06.257845 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:06.257766 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:25:06.257845 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:06.257833 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:25:06.262129 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:06.262108 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:25:06.805948 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:06.805914 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:25:06.854091 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:06.854062 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bdfdcc99b-6p8js"] Jul 09 14:25:31.877970 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:31.877930 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bdfdcc99b-6p8js" podUID="10cc1183-842f-4773-998c-ca7edf3490ed" containerName="console" containerID="cri-o://36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0" gracePeriod=15 Jul 09 14:25:32.117722 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.117700 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:25:32.213877 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.213778 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-trusted-ca-bundle\") pod \"10cc1183-842f-4773-998c-ca7edf3490ed\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " Jul 09 14:25:32.213877 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.213823 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-oauth-serving-cert\") pod \"10cc1183-842f-4773-998c-ca7edf3490ed\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " Jul 09 14:25:32.213877 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.213867 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z68fw\" (UniqueName: \"kubernetes.io/projected/10cc1183-842f-4773-998c-ca7edf3490ed-kube-api-access-z68fw\") pod \"10cc1183-842f-4773-998c-ca7edf3490ed\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " Jul 09 14:25:32.214150 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.213898 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-serving-cert\") pod \"10cc1183-842f-4773-998c-ca7edf3490ed\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " Jul 09 14:25:32.214150 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.213941 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-service-ca\") pod \"10cc1183-842f-4773-998c-ca7edf3490ed\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " Jul 09 14:25:32.214150 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.213971 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-console-config\") pod \"10cc1183-842f-4773-998c-ca7edf3490ed\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " Jul 09 14:25:32.214150 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.214007 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-oauth-config\") pod \"10cc1183-842f-4773-998c-ca7edf3490ed\" (UID: \"10cc1183-842f-4773-998c-ca7edf3490ed\") " Jul 09 14:25:32.214336 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.214222 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "10cc1183-842f-4773-998c-ca7edf3490ed" (UID: "10cc1183-842f-4773-998c-ca7edf3490ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:25:32.214394 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.214337 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "10cc1183-842f-4773-998c-ca7edf3490ed" (UID: "10cc1183-842f-4773-998c-ca7edf3490ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:25:32.214448 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.214412 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "10cc1183-842f-4773-998c-ca7edf3490ed" (UID: "10cc1183-842f-4773-998c-ca7edf3490ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:25:32.214496 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.214439 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-console-config" (OuterVolumeSpecName: "console-config") pod "10cc1183-842f-4773-998c-ca7edf3490ed" (UID: "10cc1183-842f-4773-998c-ca7edf3490ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:25:32.216082 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.216050 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "10cc1183-842f-4773-998c-ca7edf3490ed" (UID: "10cc1183-842f-4773-998c-ca7edf3490ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:25:32.216082 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.216068 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "10cc1183-842f-4773-998c-ca7edf3490ed" (UID: "10cc1183-842f-4773-998c-ca7edf3490ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:25:32.216222 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.216104 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cc1183-842f-4773-998c-ca7edf3490ed-kube-api-access-z68fw" (OuterVolumeSpecName: "kube-api-access-z68fw") pod "10cc1183-842f-4773-998c-ca7edf3490ed" (UID: "10cc1183-842f-4773-998c-ca7edf3490ed"). InnerVolumeSpecName "kube-api-access-z68fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:25:32.315121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.315089 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-trusted-ca-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:25:32.315121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.315116 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-oauth-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:25:32.315121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.315125 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68fw\" (UniqueName: \"kubernetes.io/projected/10cc1183-842f-4773-998c-ca7edf3490ed-kube-api-access-z68fw\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:25:32.315334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.315134 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:25:32.315334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.315143 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:25:32.315334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.315151 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/10cc1183-842f-4773-998c-ca7edf3490ed-console-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:25:32.315334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.315158 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/10cc1183-842f-4773-998c-ca7edf3490ed-console-oauth-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:25:32.898912 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.898841 2563 generic.go:356] "Generic (PLEG): container finished" podID="10cc1183-842f-4773-998c-ca7edf3490ed" containerID="36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0" exitCode=0 Jul 09 14:25:32.898912 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.898881 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdfdcc99b-6p8js" event={"ID":"10cc1183-842f-4773-998c-ca7edf3490ed","Type":"ContainerDied","Data":"36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0"} Jul 09 14:25:32.898912 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.898883 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdfdcc99b-6p8js" Jul 09 14:25:32.898912 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.898912 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdfdcc99b-6p8js" event={"ID":"10cc1183-842f-4773-998c-ca7edf3490ed","Type":"ContainerDied","Data":"57008db1b87ecdeb390fb5c255f7eb2bb8d2474862eb17736e4e1e7502852af6"} Jul 09 14:25:32.899373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.898932 2563 scope.go:117] "RemoveContainer" containerID="36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0" Jul 09 14:25:32.906653 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.906634 2563 scope.go:117] "RemoveContainer" containerID="36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0" Jul 09 14:25:32.906914 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:25:32.906897 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0\": container with ID starting with 36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0 not found: ID does not exist" containerID="36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0" Jul 09 14:25:32.906963 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.906922 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0"} err="failed to get container status \"36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0\": rpc error: code = NotFound desc = could not find container \"36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0\": container with ID starting with 36356adee0d6c897f6342ca49fdf105cb2765d5673c315afc22593dce48422c0 not found: ID does not exist" Jul 09 14:25:32.914906 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.914882 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bdfdcc99b-6p8js"] Jul 09 14:25:32.919190 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:32.919172 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bdfdcc99b-6p8js"] Jul 09 14:25:34.573608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:34.573566 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10cc1183-842f-4773-998c-ca7edf3490ed" path="/var/lib/kubelet/pods/10cc1183-842f-4773-998c-ca7edf3490ed/volumes" Jul 09 14:25:36.405152 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:36.405125 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:25:36.415890 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:36.415864 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:25:36.419438 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:36.419414 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:25:36.420726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:36.420708 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 14:25:36.426437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:25:36.426418 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:26:02.577682 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.577646 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk"] Jul 09 14:26:02.579096 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.577929 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10cc1183-842f-4773-998c-ca7edf3490ed" containerName="console" Jul 09 14:26:02.579096 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.577940 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10cc1183-842f-4773-998c-ca7edf3490ed" containerName="console" Jul 09 14:26:02.579096 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.578003 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10cc1183-842f-4773-998c-ca7edf3490ed" containerName="console" Jul 09 14:26:02.579922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.579903 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.585797 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.585771 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 14:26:02.585928 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.585773 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rjjv7\"" Jul 09 14:26:02.586983 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.586612 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 14:26:02.603319 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.603297 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk"] Jul 09 14:26:02.727881 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.727860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7jmx\" (UniqueName: \"kubernetes.io/projected/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-kube-api-access-p7jmx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.727998 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.727889 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.727998 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.727919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.830008 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.829931 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7jmx\" (UniqueName: \"kubernetes.io/projected/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-kube-api-access-p7jmx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.830207 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.830180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.830676 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.830648 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.830818 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.830796 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.831050 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.831024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.839295 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.839272 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7jmx\" (UniqueName: \"kubernetes.io/projected/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-kube-api-access-p7jmx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:02.889311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:02.889290 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:03.014525 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:03.014499 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk"] Jul 09 14:26:03.016737 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:26:03.016695 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fe3dd4_44db_4a38_941e_e3f935eb5fba.slice/crio-a48ff68b37e477a27cc14af5a0c2066022bf3cfb7513d821445c0cc8942cecec WatchSource:0}: Error finding container a48ff68b37e477a27cc14af5a0c2066022bf3cfb7513d821445c0cc8942cecec: Status 404 returned error can't find the container with id a48ff68b37e477a27cc14af5a0c2066022bf3cfb7513d821445c0cc8942cecec Jul 09 14:26:03.018560 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:03.018546 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:26:04.021850 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:04.021814 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" event={"ID":"a8fe3dd4-44db-4a38-941e-e3f935eb5fba","Type":"ContainerStarted","Data":"a48ff68b37e477a27cc14af5a0c2066022bf3cfb7513d821445c0cc8942cecec"} Jul 09 14:26:10.047865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:10.047830 2563 generic.go:356] "Generic (PLEG): container finished" podID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerID="8c775c81f17cce1e4388a3deb9ee13d00a3f7a3f6b9e9766a8aa398e13f5641e" exitCode=0 Jul 09 14:26:10.048203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:10.047912 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" event={"ID":"a8fe3dd4-44db-4a38-941e-e3f935eb5fba","Type":"ContainerDied","Data":"8c775c81f17cce1e4388a3deb9ee13d00a3f7a3f6b9e9766a8aa398e13f5641e"} Jul 09 14:26:13.060665 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:13.060612 2563 generic.go:356] "Generic (PLEG): container finished" podID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerID="8b56280ede39103e5642c268b990566c05c70158597e862c7242eb73c2e76e22" exitCode=0 Jul 09 14:26:13.061001 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:13.060698 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" event={"ID":"a8fe3dd4-44db-4a38-941e-e3f935eb5fba","Type":"ContainerDied","Data":"8b56280ede39103e5642c268b990566c05c70158597e862c7242eb73c2e76e22"} Jul 09 14:26:22.096705 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:22.096670 2563 generic.go:356] "Generic (PLEG): container finished" podID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerID="016ceed67df8e37a175ec52113792ff4c16d7a66bedbb07722d073fd9a5633fd" exitCode=0 Jul 09 14:26:22.097051 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:22.096749 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" event={"ID":"a8fe3dd4-44db-4a38-941e-e3f935eb5fba","Type":"ContainerDied","Data":"016ceed67df8e37a175ec52113792ff4c16d7a66bedbb07722d073fd9a5633fd"} Jul 09 14:26:23.217881 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.217857 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:23.287915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.287887 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-util\") pod \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " Jul 09 14:26:23.288029 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.287931 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7jmx\" (UniqueName: \"kubernetes.io/projected/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-kube-api-access-p7jmx\") pod \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " Jul 09 14:26:23.288029 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.287977 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-bundle\") pod \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\" (UID: \"a8fe3dd4-44db-4a38-941e-e3f935eb5fba\") " Jul 09 14:26:23.288599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.288577 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-bundle" (OuterVolumeSpecName: "bundle") pod "a8fe3dd4-44db-4a38-941e-e3f935eb5fba" (UID: "a8fe3dd4-44db-4a38-941e-e3f935eb5fba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:26:23.290020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.289999 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-kube-api-access-p7jmx" (OuterVolumeSpecName: "kube-api-access-p7jmx") pod "a8fe3dd4-44db-4a38-941e-e3f935eb5fba" (UID: "a8fe3dd4-44db-4a38-941e-e3f935eb5fba"). InnerVolumeSpecName "kube-api-access-p7jmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:26:23.291877 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.291858 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-util" (OuterVolumeSpecName: "util") pod "a8fe3dd4-44db-4a38-941e-e3f935eb5fba" (UID: "a8fe3dd4-44db-4a38-941e-e3f935eb5fba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:26:23.388867 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.388787 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-util\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:26:23.388867 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.388817 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7jmx\" (UniqueName: \"kubernetes.io/projected/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-kube-api-access-p7jmx\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:26:23.388867 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:23.388828 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8fe3dd4-44db-4a38-941e-e3f935eb5fba-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:26:24.106388 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:24.106355 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" event={"ID":"a8fe3dd4-44db-4a38-941e-e3f935eb5fba","Type":"ContainerDied","Data":"a48ff68b37e477a27cc14af5a0c2066022bf3cfb7513d821445c0cc8942cecec"} Jul 09 14:26:24.106388 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:24.106373 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ddzrk" Jul 09 14:26:24.106586 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:24.106404 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48ff68b37e477a27cc14af5a0c2066022bf3cfb7513d821445c0cc8942cecec" Jul 09 14:26:30.749673 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.749575 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6"] Jul 09 14:26:30.750086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.749896 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerName="util" Jul 09 14:26:30.750086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.749911 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerName="util" Jul 09 14:26:30.750086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.749922 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerName="extract" Jul 09 14:26:30.750086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.749928 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerName="extract" Jul 09 14:26:30.750086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.749937 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerName="pull" Jul 09 14:26:30.750086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.749943 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerName="pull" Jul 09 14:26:30.750086 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.750003 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8fe3dd4-44db-4a38-941e-e3f935eb5fba" containerName="extract" Jul 09 14:26:30.751830 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.751812 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:30.757970 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.757467 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 14:26:30.757970 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.757947 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:26:30.758102 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.757999 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8p7z7\"" Jul 09 14:26:30.777188 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.777166 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6"] Jul 09 14:26:30.841151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.841129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x5h8\" (UniqueName: \"kubernetes.io/projected/a9ab905c-d540-4f28-a29f-99f1d0febc85-kube-api-access-4x5h8\") pod \"cert-manager-operator-controller-manager-55c444bf86-clbx6\" (UID: \"a9ab905c-d540-4f28-a29f-99f1d0febc85\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:30.841251 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.841180 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9ab905c-d540-4f28-a29f-99f1d0febc85-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-clbx6\" (UID: \"a9ab905c-d540-4f28-a29f-99f1d0febc85\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:30.941577 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.941553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x5h8\" (UniqueName: \"kubernetes.io/projected/a9ab905c-d540-4f28-a29f-99f1d0febc85-kube-api-access-4x5h8\") pod \"cert-manager-operator-controller-manager-55c444bf86-clbx6\" (UID: \"a9ab905c-d540-4f28-a29f-99f1d0febc85\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:30.941685 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.941612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9ab905c-d540-4f28-a29f-99f1d0febc85-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-clbx6\" (UID: \"a9ab905c-d540-4f28-a29f-99f1d0febc85\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:30.941950 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.941933 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9ab905c-d540-4f28-a29f-99f1d0febc85-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-clbx6\" (UID: \"a9ab905c-d540-4f28-a29f-99f1d0febc85\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:30.954648 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:30.954610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x5h8\" (UniqueName: \"kubernetes.io/projected/a9ab905c-d540-4f28-a29f-99f1d0febc85-kube-api-access-4x5h8\") pod \"cert-manager-operator-controller-manager-55c444bf86-clbx6\" (UID: \"a9ab905c-d540-4f28-a29f-99f1d0febc85\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:31.060690 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:31.060668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" Jul 09 14:26:31.198510 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:31.198483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6"] Jul 09 14:26:31.202360 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:26:31.202337 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ab905c_d540_4f28_a29f_99f1d0febc85.slice/crio-4af8f7a4fa4cb15adbd1eaeea47dde8824998a1dee3852b1c9618b9ad354045f WatchSource:0}: Error finding container 4af8f7a4fa4cb15adbd1eaeea47dde8824998a1dee3852b1c9618b9ad354045f: Status 404 returned error can't find the container with id 4af8f7a4fa4cb15adbd1eaeea47dde8824998a1dee3852b1c9618b9ad354045f Jul 09 14:26:32.138453 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:32.138415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" event={"ID":"a9ab905c-d540-4f28-a29f-99f1d0febc85","Type":"ContainerStarted","Data":"4af8f7a4fa4cb15adbd1eaeea47dde8824998a1dee3852b1c9618b9ad354045f"} Jul 09 14:26:34.148169 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.148136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" event={"ID":"a9ab905c-d540-4f28-a29f-99f1d0febc85","Type":"ContainerStarted","Data":"f394b640d055170a33d731db27c73384d93efadd4386ebd3388865b8c267dd83"} Jul 09 14:26:34.170899 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.170853 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-clbx6" podStartSLOduration=2.221644124 podStartE2EDuration="4.17083887s" totalImagesPullingTime="1.949194746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:26:34.169852036 +0000 UTC m=+358.190935292" watchObservedRunningTime="2026-07-09 14:26:34.17083887 +0000 UTC m=+358.191922134" Jul 09 14:26:34.934218 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.934185 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn"] Jul 09 14:26:34.936804 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.936787 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:34.941079 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.941061 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rjjv7\"" Jul 09 14:26:34.941527 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.941515 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 14:26:34.942017 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.942001 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 14:26:34.961168 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.961149 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn"] Jul 09 14:26:34.971722 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.971703 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:34.971804 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.971732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:34.971804 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:34.971756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxmg\" (UniqueName: \"kubernetes.io/projected/fe147065-89ee-45e2-89b1-82638d709c74-kube-api-access-2rxmg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.072682 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.072656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.072788 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.072686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.072788 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.072711 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxmg\" (UniqueName: \"kubernetes.io/projected/fe147065-89ee-45e2-89b1-82638d709c74-kube-api-access-2rxmg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.073036 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.073018 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.073076 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.073043 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.099506 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.099477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxmg\" (UniqueName: \"kubernetes.io/projected/fe147065-89ee-45e2-89b1-82638d709c74-kube-api-access-2rxmg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.179917 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.179895 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p56ng"] Jul 09 14:26:35.182386 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.182366 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.184929 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.184882 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 14:26:35.185020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.184946 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 14:26:35.185078 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.185036 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-47dnd\"" Jul 09 14:26:35.195042 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.195022 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p56ng"] Jul 09 14:26:35.245093 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.245075 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:35.274527 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.274506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p56ng\" (UID: \"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.274653 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.274538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67z6b\" (UniqueName: \"kubernetes.io/projected/fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f-kube-api-access-67z6b\") pod \"cert-manager-webhook-6486ffd997-p56ng\" (UID: \"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.374048 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.374009 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn"] Jul 09 14:26:35.375569 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.375545 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p56ng\" (UID: \"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.375706 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.375581 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67z6b\" (UniqueName: \"kubernetes.io/projected/fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f-kube-api-access-67z6b\") pod \"cert-manager-webhook-6486ffd997-p56ng\" (UID: \"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.376729 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:26:35.376708 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe147065_89ee_45e2_89b1_82638d709c74.slice/crio-6082090ead6cbeaf1cabb071bf4872486bb7a2fac36a056393df262246bc3863 WatchSource:0}: Error finding container 6082090ead6cbeaf1cabb071bf4872486bb7a2fac36a056393df262246bc3863: Status 404 returned error can't find the container with id 6082090ead6cbeaf1cabb071bf4872486bb7a2fac36a056393df262246bc3863 Jul 09 14:26:35.384437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.384411 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-p56ng\" (UID: \"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.384549 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.384506 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67z6b\" (UniqueName: \"kubernetes.io/projected/fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f-kube-api-access-67z6b\") pod \"cert-manager-webhook-6486ffd997-p56ng\" (UID: \"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.503849 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.503823 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:35.626889 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:35.626863 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-p56ng"] Jul 09 14:26:35.629356 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:26:35.629332 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbc49a0c_e7a0_4d01_b017_b4d2d383eb8f.slice/crio-3b3afd85e27b184f4b3832e17025ac1213a6c64f8e0ccef83cf965059a0b0d01 WatchSource:0}: Error finding container 3b3afd85e27b184f4b3832e17025ac1213a6c64f8e0ccef83cf965059a0b0d01: Status 404 returned error can't find the container with id 3b3afd85e27b184f4b3832e17025ac1213a6c64f8e0ccef83cf965059a0b0d01 Jul 09 14:26:36.156668 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:36.156634 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" event={"ID":"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f","Type":"ContainerStarted","Data":"3b3afd85e27b184f4b3832e17025ac1213a6c64f8e0ccef83cf965059a0b0d01"} Jul 09 14:26:36.158142 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:36.158116 2563 generic.go:356] "Generic (PLEG): container finished" podID="fe147065-89ee-45e2-89b1-82638d709c74" containerID="4ed7ff152c0d3621a1e8821304acf24565db3ce32ace9db8ec62ef608a7339b8" exitCode=0 Jul 09 14:26:36.158260 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:36.158175 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" event={"ID":"fe147065-89ee-45e2-89b1-82638d709c74","Type":"ContainerDied","Data":"4ed7ff152c0d3621a1e8821304acf24565db3ce32ace9db8ec62ef608a7339b8"} Jul 09 14:26:36.158260 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:36.158224 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" event={"ID":"fe147065-89ee-45e2-89b1-82638d709c74","Type":"ContainerStarted","Data":"6082090ead6cbeaf1cabb071bf4872486bb7a2fac36a056393df262246bc3863"} Jul 09 14:26:39.174809 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:39.174769 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" event={"ID":"fbc49a0c-e7a0-4d01-b017-b4d2d383eb8f","Type":"ContainerStarted","Data":"945561f0ab2002a9c903cbd9ec3c9af0f531f526c39d7a9be122be0bb9ca4768"} Jul 09 14:26:39.175146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:39.174824 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:39.176562 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:39.176534 2563 generic.go:356] "Generic (PLEG): container finished" podID="fe147065-89ee-45e2-89b1-82638d709c74" containerID="eeb2d5ee118bb19a453dcf58020f832a5c90004128f0f30790a60d114a5004ba" exitCode=0 Jul 09 14:26:39.176673 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:39.176585 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" event={"ID":"fe147065-89ee-45e2-89b1-82638d709c74","Type":"ContainerDied","Data":"eeb2d5ee118bb19a453dcf58020f832a5c90004128f0f30790a60d114a5004ba"} Jul 09 14:26:39.191054 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:39.190990 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" podStartSLOduration=0.822966852 podStartE2EDuration="4.190973415s" totalImagesPullingTime="3.368006563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:26:39.189929493 +0000 UTC m=+363.211012757" watchObservedRunningTime="2026-07-09 14:26:39.190973415 +0000 UTC m=+363.212056681" Jul 09 14:26:40.182633 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:40.182589 2563 generic.go:356] "Generic (PLEG): container finished" podID="fe147065-89ee-45e2-89b1-82638d709c74" containerID="893d7237ba272c125282f7f2dbd2d0231e20b541c2b7e0d0c7d47424b6e15abd" exitCode=0 Jul 09 14:26:40.182985 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:40.182721 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" event={"ID":"fe147065-89ee-45e2-89b1-82638d709c74","Type":"ContainerDied","Data":"893d7237ba272c125282f7f2dbd2d0231e20b541c2b7e0d0c7d47424b6e15abd"} Jul 09 14:26:41.313309 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.313286 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:41.428229 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.428209 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-util\") pod \"fe147065-89ee-45e2-89b1-82638d709c74\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " Jul 09 14:26:41.428341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.428241 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-bundle\") pod \"fe147065-89ee-45e2-89b1-82638d709c74\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " Jul 09 14:26:41.428341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.428270 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rxmg\" (UniqueName: \"kubernetes.io/projected/fe147065-89ee-45e2-89b1-82638d709c74-kube-api-access-2rxmg\") pod \"fe147065-89ee-45e2-89b1-82638d709c74\" (UID: \"fe147065-89ee-45e2-89b1-82638d709c74\") " Jul 09 14:26:41.428635 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.428597 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-bundle" (OuterVolumeSpecName: "bundle") pod "fe147065-89ee-45e2-89b1-82638d709c74" (UID: "fe147065-89ee-45e2-89b1-82638d709c74"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:26:41.430236 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.430216 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe147065-89ee-45e2-89b1-82638d709c74-kube-api-access-2rxmg" (OuterVolumeSpecName: "kube-api-access-2rxmg") pod "fe147065-89ee-45e2-89b1-82638d709c74" (UID: "fe147065-89ee-45e2-89b1-82638d709c74"). InnerVolumeSpecName "kube-api-access-2rxmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:26:41.432096 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.432068 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-util" (OuterVolumeSpecName: "util") pod "fe147065-89ee-45e2-89b1-82638d709c74" (UID: "fe147065-89ee-45e2-89b1-82638d709c74"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:26:41.529264 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.529244 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:26:41.529264 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.529262 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rxmg\" (UniqueName: \"kubernetes.io/projected/fe147065-89ee-45e2-89b1-82638d709c74-kube-api-access-2rxmg\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:26:41.529377 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:41.529271 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe147065-89ee-45e2-89b1-82638d709c74-util\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:26:42.193637 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:42.193600 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" Jul 09 14:26:42.193806 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:42.193600 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ffvlfn" event={"ID":"fe147065-89ee-45e2-89b1-82638d709c74","Type":"ContainerDied","Data":"6082090ead6cbeaf1cabb071bf4872486bb7a2fac36a056393df262246bc3863"} Jul 09 14:26:42.193806 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:42.193746 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6082090ead6cbeaf1cabb071bf4872486bb7a2fac36a056393df262246bc3863" Jul 09 14:26:45.184861 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:45.184824 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-p56ng" Jul 09 14:26:51.683054 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683027 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx"] Jul 09 14:26:51.683451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683313 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe147065-89ee-45e2-89b1-82638d709c74" containerName="pull" Jul 09 14:26:51.683451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683323 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe147065-89ee-45e2-89b1-82638d709c74" containerName="pull" Jul 09 14:26:51.683451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683337 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe147065-89ee-45e2-89b1-82638d709c74" containerName="util" Jul 09 14:26:51.683451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683343 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe147065-89ee-45e2-89b1-82638d709c74" containerName="util" Jul 09 14:26:51.683451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683354 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe147065-89ee-45e2-89b1-82638d709c74" containerName="extract" Jul 09 14:26:51.683451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683361 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe147065-89ee-45e2-89b1-82638d709c74" containerName="extract" Jul 09 14:26:51.683451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.683419 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe147065-89ee-45e2-89b1-82638d709c74" containerName="extract" Jul 09 14:26:51.716931 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.716905 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:51.717060 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.716909 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx"] Jul 09 14:26:51.719810 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.719783 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 14:26:51.719810 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.719808 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 14:26:51.719963 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.719827 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qzn2x\"" Jul 09 14:26:51.807593 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.807566 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdfsp\" (UniqueName: \"kubernetes.io/projected/e761aa26-971b-44b8-95ee-503f96dc06b9-kube-api-access-zdfsp\") pod \"openshift-lws-operator-fd8ccff4c-2k8zx\" (UID: \"e761aa26-971b-44b8-95ee-503f96dc06b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:51.807755 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.807660 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e761aa26-971b-44b8-95ee-503f96dc06b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2k8zx\" (UID: \"e761aa26-971b-44b8-95ee-503f96dc06b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:51.908046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.908021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfsp\" (UniqueName: \"kubernetes.io/projected/e761aa26-971b-44b8-95ee-503f96dc06b9-kube-api-access-zdfsp\") pod \"openshift-lws-operator-fd8ccff4c-2k8zx\" (UID: \"e761aa26-971b-44b8-95ee-503f96dc06b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:51.908176 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.908067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e761aa26-971b-44b8-95ee-503f96dc06b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2k8zx\" (UID: \"e761aa26-971b-44b8-95ee-503f96dc06b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:51.908335 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.908316 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e761aa26-971b-44b8-95ee-503f96dc06b9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2k8zx\" (UID: \"e761aa26-971b-44b8-95ee-503f96dc06b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:51.917091 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:51.917069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfsp\" (UniqueName: \"kubernetes.io/projected/e761aa26-971b-44b8-95ee-503f96dc06b9-kube-api-access-zdfsp\") pod \"openshift-lws-operator-fd8ccff4c-2k8zx\" (UID: \"e761aa26-971b-44b8-95ee-503f96dc06b9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:52.026288 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:52.026266 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" Jul 09 14:26:52.168371 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:52.168350 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx"] Jul 09 14:26:52.170487 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:26:52.170456 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode761aa26_971b_44b8_95ee_503f96dc06b9.slice/crio-e1420a9c0fc7af039f5805315e284aa627691fa3245c4fa77b8d506a9549fed3 WatchSource:0}: Error finding container e1420a9c0fc7af039f5805315e284aa627691fa3245c4fa77b8d506a9549fed3: Status 404 returned error can't find the container with id e1420a9c0fc7af039f5805315e284aa627691fa3245c4fa77b8d506a9549fed3 Jul 09 14:26:52.235484 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:52.235455 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" event={"ID":"e761aa26-971b-44b8-95ee-503f96dc06b9","Type":"ContainerStarted","Data":"e1420a9c0fc7af039f5805315e284aa627691fa3245c4fa77b8d506a9549fed3"} Jul 09 14:26:56.253394 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:56.253322 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" event={"ID":"e761aa26-971b-44b8-95ee-503f96dc06b9","Type":"ContainerStarted","Data":"43ddb2f615cdb40e21405ec6085091e6657ae80319b8ed004ba8a48c07697d6f"} Jul 09 14:26:56.269844 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:26:56.269803 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2k8zx" podStartSLOduration=1.536715901 podStartE2EDuration="5.26978906s" totalImagesPullingTime="3.733073159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:26:56.268563723 +0000 UTC m=+380.289646988" watchObservedRunningTime="2026-07-09 14:26:56.26978906 +0000 UTC m=+380.290872323" Jul 09 14:27:01.541436 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.541401 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x"] Jul 09 14:27:01.543814 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.543798 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.546498 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.546476 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 14:27:01.546639 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.546594 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rjjv7\"" Jul 09 14:27:01.547401 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.547386 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 14:27:01.553515 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.553495 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x"] Jul 09 14:27:01.681434 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.681398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v66vj\" (UniqueName: \"kubernetes.io/projected/4b3f881b-df58-4586-a722-d047394c3638-kube-api-access-v66vj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.681610 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.681475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.681610 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.681530 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.782545 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.782514 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.782707 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.782559 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.782707 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.782599 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v66vj\" (UniqueName: \"kubernetes.io/projected/4b3f881b-df58-4586-a722-d047394c3638-kube-api-access-v66vj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.782997 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.782976 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.783039 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.782986 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.791376 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.791343 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v66vj\" (UniqueName: \"kubernetes.io/projected/4b3f881b-df58-4586-a722-d047394c3638-kube-api-access-v66vj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.853001 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.852943 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:01.982414 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:01.982391 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x"] Jul 09 14:27:02.280808 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:02.280775 2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3f881b-df58-4586-a722-d047394c3638" containerID="ee3f1c7e539a0d5ef90bdc4111316bfde3fd25b78035ac09ad188015bc3479e5" exitCode=0 Jul 09 14:27:02.280982 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:02.280864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" event={"ID":"4b3f881b-df58-4586-a722-d047394c3638","Type":"ContainerDied","Data":"ee3f1c7e539a0d5ef90bdc4111316bfde3fd25b78035ac09ad188015bc3479e5"} Jul 09 14:27:02.280982 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:02.280928 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" event={"ID":"4b3f881b-df58-4586-a722-d047394c3638","Type":"ContainerStarted","Data":"6588270b9488ae7b26c25fe5c1f255f2e8836bf8392b14f26bdae6e5616f2488"} Jul 09 14:27:03.287213 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:03.287177 2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3f881b-df58-4586-a722-d047394c3638" containerID="778e2b85294d67e318d6225057a286e643cad1272b7bea0b7bb1eea5a46469f7" exitCode=0 Jul 09 14:27:03.287664 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:03.287256 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" event={"ID":"4b3f881b-df58-4586-a722-d047394c3638","Type":"ContainerDied","Data":"778e2b85294d67e318d6225057a286e643cad1272b7bea0b7bb1eea5a46469f7"} Jul 09 14:27:04.293663 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:04.293604 2563 generic.go:356] "Generic (PLEG): container finished" podID="4b3f881b-df58-4586-a722-d047394c3638" containerID="6cffa05f13eeb15967cf50adc1287e9ba614f341c7b2aafa3a183ea123a354d2" exitCode=0 Jul 09 14:27:04.294046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:04.293663 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" event={"ID":"4b3f881b-df58-4586-a722-d047394c3638","Type":"ContainerDied","Data":"6cffa05f13eeb15967cf50adc1287e9ba614f341c7b2aafa3a183ea123a354d2"} Jul 09 14:27:05.423588 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.423567 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:05.515310 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.515287 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v66vj\" (UniqueName: \"kubernetes.io/projected/4b3f881b-df58-4586-a722-d047394c3638-kube-api-access-v66vj\") pod \"4b3f881b-df58-4586-a722-d047394c3638\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " Jul 09 14:27:05.515410 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.515345 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-bundle\") pod \"4b3f881b-df58-4586-a722-d047394c3638\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " Jul 09 14:27:05.515410 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.515403 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-util\") pod \"4b3f881b-df58-4586-a722-d047394c3638\" (UID: \"4b3f881b-df58-4586-a722-d047394c3638\") " Jul 09 14:27:05.516083 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.516053 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-bundle" (OuterVolumeSpecName: "bundle") pod "4b3f881b-df58-4586-a722-d047394c3638" (UID: "4b3f881b-df58-4586-a722-d047394c3638"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:27:05.517369 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.517347 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3f881b-df58-4586-a722-d047394c3638-kube-api-access-v66vj" (OuterVolumeSpecName: "kube-api-access-v66vj") pod "4b3f881b-df58-4586-a722-d047394c3638" (UID: "4b3f881b-df58-4586-a722-d047394c3638"). InnerVolumeSpecName "kube-api-access-v66vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:27:05.521050 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.521031 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-util" (OuterVolumeSpecName: "util") pod "4b3f881b-df58-4586-a722-d047394c3638" (UID: "4b3f881b-df58-4586-a722-d047394c3638"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:27:05.616875 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.616825 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v66vj\" (UniqueName: \"kubernetes.io/projected/4b3f881b-df58-4586-a722-d047394c3638-kube-api-access-v66vj\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:27:05.616875 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.616846 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:27:05.616875 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:05.616855 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b3f881b-df58-4586-a722-d047394c3638-util\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:27:06.305922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:06.305894 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" Jul 09 14:27:06.306059 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:06.305906 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqsm6x" event={"ID":"4b3f881b-df58-4586-a722-d047394c3638","Type":"ContainerDied","Data":"6588270b9488ae7b26c25fe5c1f255f2e8836bf8392b14f26bdae6e5616f2488"} Jul 09 14:27:06.306059 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:06.305974 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6588270b9488ae7b26c25fe5c1f255f2e8836bf8392b14f26bdae6e5616f2488" Jul 09 14:27:14.478827 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.478791 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74"] Jul 09 14:27:14.479206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.479133 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3f881b-df58-4586-a722-d047394c3638" containerName="extract" Jul 09 14:27:14.479206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.479147 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3f881b-df58-4586-a722-d047394c3638" containerName="extract" Jul 09 14:27:14.479206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.479162 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3f881b-df58-4586-a722-d047394c3638" containerName="util" Jul 09 14:27:14.479206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.479171 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3f881b-df58-4586-a722-d047394c3638" containerName="util" Jul 09 14:27:14.479206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.479179 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b3f881b-df58-4586-a722-d047394c3638" containerName="pull" Jul 09 14:27:14.479206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.479187 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b3f881b-df58-4586-a722-d047394c3638" containerName="pull" Jul 09 14:27:14.479391 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.479278 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b3f881b-df58-4586-a722-d047394c3638" containerName="extract" Jul 09 14:27:14.481512 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.481482 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.484419 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.484397 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 14:27:14.484850 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.484820 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 14:27:14.484943 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.484903 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 14:27:14.484943 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.484915 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 14:27:14.484943 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.484929 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fdgwt\"" Jul 09 14:27:14.508048 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.508026 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74"] Jul 09 14:27:14.582364 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.582336 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/396125ee-ee41-419c-87ff-a5e807f6079b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.582467 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.582370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396125ee-ee41-419c-87ff-a5e807f6079b-tmp\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.582467 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.582398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/396125ee-ee41-419c-87ff-a5e807f6079b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.582577 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.582484 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4ww\" (UniqueName: \"kubernetes.io/projected/396125ee-ee41-419c-87ff-a5e807f6079b-kube-api-access-qj4ww\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.582577 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.582515 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/396125ee-ee41-419c-87ff-a5e807f6079b-manifests\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.683399 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.683374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/396125ee-ee41-419c-87ff-a5e807f6079b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.683504 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.683406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396125ee-ee41-419c-87ff-a5e807f6079b-tmp\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.683504 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.683434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/396125ee-ee41-419c-87ff-a5e807f6079b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.683504 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.683476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4ww\" (UniqueName: \"kubernetes.io/projected/396125ee-ee41-419c-87ff-a5e807f6079b-kube-api-access-qj4ww\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.683676 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.683513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/396125ee-ee41-419c-87ff-a5e807f6079b-manifests\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.683887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.683868 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/396125ee-ee41-419c-87ff-a5e807f6079b-manifests\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.685698 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.685676 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/396125ee-ee41-419c-87ff-a5e807f6079b-tmp\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.685816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.685798 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/396125ee-ee41-419c-87ff-a5e807f6079b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.685924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.685906 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/396125ee-ee41-419c-87ff-a5e807f6079b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.691899 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.691879 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4ww\" (UniqueName: \"kubernetes.io/projected/396125ee-ee41-419c-87ff-a5e807f6079b-kube-api-access-qj4ww\") pod \"opendatahub-operator-controller-manager-c7757d956-ngt74\" (UID: \"396125ee-ee41-419c-87ff-a5e807f6079b\") " pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.792369 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.792348 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:14.944664 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:14.944634 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74"] Jul 09 14:27:14.947677 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:27:14.947643 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396125ee_ee41_419c_87ff_a5e807f6079b.slice/crio-54f70a44db260204c2cbe1ae7b298e71eb992e250e7889c010b43104ddb2abfd WatchSource:0}: Error finding container 54f70a44db260204c2cbe1ae7b298e71eb992e250e7889c010b43104ddb2abfd: Status 404 returned error can't find the container with id 54f70a44db260204c2cbe1ae7b298e71eb992e250e7889c010b43104ddb2abfd Jul 09 14:27:15.349869 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:15.349835 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" event={"ID":"396125ee-ee41-419c-87ff-a5e807f6079b","Type":"ContainerStarted","Data":"54f70a44db260204c2cbe1ae7b298e71eb992e250e7889c010b43104ddb2abfd"} Jul 09 14:27:18.365576 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:18.365496 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" event={"ID":"396125ee-ee41-419c-87ff-a5e807f6079b","Type":"ContainerStarted","Data":"1af622ad4ed3cf5724397e706ff9bf333061025cc39301e4317f43108e580b7d"} Jul 09 14:27:19.371789 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:19.371755 2563 generic.go:356] "Generic (PLEG): container finished" podID="396125ee-ee41-419c-87ff-a5e807f6079b" containerID="1af622ad4ed3cf5724397e706ff9bf333061025cc39301e4317f43108e580b7d" exitCode=0 Jul 09 14:27:19.372117 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:19.371804 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" event={"ID":"396125ee-ee41-419c-87ff-a5e807f6079b","Type":"ContainerDied","Data":"1af622ad4ed3cf5724397e706ff9bf333061025cc39301e4317f43108e580b7d"} Jul 09 14:27:20.378505 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:20.378462 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" event={"ID":"396125ee-ee41-419c-87ff-a5e807f6079b","Type":"ContainerStarted","Data":"fe908f5fbe766816f3a76697db21d8656e428f12bc6f601148a4b538d43eee6b"} Jul 09 14:27:20.378863 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:20.378598 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:20.431145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:20.431102 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" podStartSLOduration=1.66374399 podStartE2EDuration="6.431087637s" totalImagesPullingTime="3.726987767s" totalInitContainerRuntime="1.04035588s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:27:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:27:20.430095063 +0000 UTC m=+404.451178327" watchObservedRunningTime="2026-07-09 14:27:20.431087637 +0000 UTC m=+404.452170899" Jul 09 14:27:31.385051 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.385008 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c7757d956-ngt74" Jul 09 14:27:31.778231 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.778197 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6"] Jul 09 14:27:31.784585 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.784564 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:31.787384 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.787363 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 14:27:31.787488 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.787374 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 14:27:31.787548 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.787524 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7ktkf\"" Jul 09 14:27:31.787548 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.787537 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 14:27:31.792767 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.792749 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6"] Jul 09 14:27:31.909243 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.909216 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7594106-a2f2-44ba-9d91-40d12f177f95-metrics-cert\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:31.909353 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.909250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8r8\" (UniqueName: \"kubernetes.io/projected/c7594106-a2f2-44ba-9d91-40d12f177f95-kube-api-access-7z8r8\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:31.909353 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.909337 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c7594106-a2f2-44ba-9d91-40d12f177f95-manager-config\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:31.909426 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:31.909354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7594106-a2f2-44ba-9d91-40d12f177f95-cert\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.009855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.009819 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c7594106-a2f2-44ba-9d91-40d12f177f95-manager-config\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.009855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.009856 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7594106-a2f2-44ba-9d91-40d12f177f95-cert\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.010084 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.009888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7594106-a2f2-44ba-9d91-40d12f177f95-metrics-cert\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.010084 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.009913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8r8\" (UniqueName: \"kubernetes.io/projected/c7594106-a2f2-44ba-9d91-40d12f177f95-kube-api-access-7z8r8\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.010558 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.010534 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c7594106-a2f2-44ba-9d91-40d12f177f95-manager-config\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.012322 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.012299 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7594106-a2f2-44ba-9d91-40d12f177f95-cert\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.012403 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.012338 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7594106-a2f2-44ba-9d91-40d12f177f95-metrics-cert\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.018201 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.018180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8r8\" (UniqueName: \"kubernetes.io/projected/c7594106-a2f2-44ba-9d91-40d12f177f95-kube-api-access-7z8r8\") pod \"lws-controller-manager-7988959d95-v4gs6\" (UID: \"c7594106-a2f2-44ba-9d91-40d12f177f95\") " pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.094062 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.093983 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:32.244310 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.244258 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6"] Jul 09 14:27:32.246380 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:27:32.246355 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7594106_a2f2_44ba_9d91_40d12f177f95.slice/crio-ddd138569bb3a9069aa64394e94a28ec6031a6f99c19911e546a8ced45ddc28c WatchSource:0}: Error finding container ddd138569bb3a9069aa64394e94a28ec6031a6f99c19911e546a8ced45ddc28c: Status 404 returned error can't find the container with id ddd138569bb3a9069aa64394e94a28ec6031a6f99c19911e546a8ced45ddc28c Jul 09 14:27:32.436930 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:32.436846 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" event={"ID":"c7594106-a2f2-44ba-9d91-40d12f177f95","Type":"ContainerStarted","Data":"ddd138569bb3a9069aa64394e94a28ec6031a6f99c19911e546a8ced45ddc28c"} Jul 09 14:27:34.447536 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:34.447494 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" event={"ID":"c7594106-a2f2-44ba-9d91-40d12f177f95","Type":"ContainerStarted","Data":"64d118cab521e7f169a9562593aece100671e8dec0ee2514cd138c1086e59b6a"} Jul 09 14:27:34.448011 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:34.447593 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:34.496039 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:34.495986 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" podStartSLOduration=2.103129618 podStartE2EDuration="3.495970222s" totalImagesPullingTime="1.392840604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:27:34.490914226 +0000 UTC m=+418.511997490" watchObservedRunningTime="2026-07-09 14:27:34.495970222 +0000 UTC m=+418.517053485" Jul 09 14:27:43.351278 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.351247 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx"] Jul 09 14:27:43.354605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.354589 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.358128 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.358077 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7zfm5\"" Jul 09 14:27:43.358321 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.358300 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 14:27:43.372034 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.372012 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx"] Jul 09 14:27:43.389798 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.389769 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.389926 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.389827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.389926 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.389869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.390060 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.389949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.390060 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.390037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9vb\" (UniqueName: \"kubernetes.io/projected/41f155d6-09a0-4936-9a7d-8f1458ba9685-kube-api-access-7l9vb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.390164 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.390068 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.390164 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.390114 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f155d6-09a0-4936-9a7d-8f1458ba9685-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.390233 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.390175 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.390265 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.390249 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.490744 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9vb\" (UniqueName: \"kubernetes.io/projected/41f155d6-09a0-4936-9a7d-8f1458ba9685-kube-api-access-7l9vb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.490859 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.490859 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490773 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f155d6-09a0-4936-9a7d-8f1458ba9685-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.490859 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491023 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490869 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491023 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491023 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491023 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491023 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.490988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491363 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.491340 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491423 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.491381 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.491437 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.491467 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f155d6-09a0-4936-9a7d-8f1458ba9685-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.491486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.491476 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.493202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.493181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.493469 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.493449 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.499076 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.499052 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f155d6-09a0-4936-9a7d-8f1458ba9685-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.500297 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.500274 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9vb\" (UniqueName: \"kubernetes.io/projected/41f155d6-09a0-4936-9a7d-8f1458ba9685-kube-api-access-7l9vb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42cjjx\" (UID: \"41f155d6-09a0-4936-9a7d-8f1458ba9685\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.633607 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.633545 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf"] Jul 09 14:27:43.636127 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.636113 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.639118 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.639102 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8tbbg\"" Jul 09 14:27:43.639256 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.639242 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 14:27:43.639657 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.639642 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 14:27:43.640197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.640184 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 14:27:43.664745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.664726 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:43.675295 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.675268 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf"] Jul 09 14:27:43.692350 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.692321 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.692444 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.692361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.692444 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.692428 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.692521 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.692467 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4m5v\" (UniqueName: \"kubernetes.io/projected/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-kube-api-access-v4m5v\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.692521 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.692496 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.692521 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.692515 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.692651 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.692629 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.793474 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.793441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.793644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.793491 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.793644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.793534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4m5v\" (UniqueName: \"kubernetes.io/projected/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-kube-api-access-v4m5v\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.793644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.793575 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.793644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.793598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.793870 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.793683 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.793870 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.793742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.794204 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.794181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.795793 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.795771 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.795973 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.795951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.796194 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.796180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.796239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.796227 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.804223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.804120 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx"] Jul 09 14:27:43.806598 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:27:43.806578 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f155d6_09a0_4936_9a7d_8f1458ba9685.slice/crio-bbf1f7e8b337b788e37809adc77b913df5a3a86dd90c8bdaf1c7ef0ff10b5701 WatchSource:0}: Error finding container bbf1f7e8b337b788e37809adc77b913df5a3a86dd90c8bdaf1c7ef0ff10b5701: Status 404 returned error can't find the container with id bbf1f7e8b337b788e37809adc77b913df5a3a86dd90c8bdaf1c7ef0ff10b5701 Jul 09 14:27:43.806713 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.806704 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.810958 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.810940 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4m5v\" (UniqueName: \"kubernetes.io/projected/f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef-kube-api-access-v4m5v\") pod \"istiod-openshift-gateway-94bb8fbfd-tvnvf\" (UID: \"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:43.945264 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:43.945215 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:44.103239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:44.103213 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf"] Jul 09 14:27:44.104662 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:27:44.104635 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f09d0e_a4d0_4463_9bcb_c41430ebe4ef.slice/crio-eba7ac0f4d1657e5905f510d20ed9295c0cccc53a9e097e95c852f2563d9bf89 WatchSource:0}: Error finding container eba7ac0f4d1657e5905f510d20ed9295c0cccc53a9e097e95c852f2563d9bf89: Status 404 returned error can't find the container with id eba7ac0f4d1657e5905f510d20ed9295c0cccc53a9e097e95c852f2563d9bf89 Jul 09 14:27:44.492866 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:44.492835 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" event={"ID":"41f155d6-09a0-4936-9a7d-8f1458ba9685","Type":"ContainerStarted","Data":"bbf1f7e8b337b788e37809adc77b913df5a3a86dd90c8bdaf1c7ef0ff10b5701"} Jul 09 14:27:44.493914 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:44.493892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" event={"ID":"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef","Type":"ContainerStarted","Data":"eba7ac0f4d1657e5905f510d20ed9295c0cccc53a9e097e95c852f2563d9bf89"} Jul 09 14:27:45.455433 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:45.455399 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7988959d95-v4gs6" Jul 09 14:27:47.624390 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:47.624354 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:27:47.624782 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:47.624434 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:27:47.628514 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:47.628489 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:27:47.628571 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:47.628546 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:27:47.628611 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:47.628574 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:27:48.516944 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.516907 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" event={"ID":"f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef","Type":"ContainerStarted","Data":"252232f1b3b97b7ea365e4d4bd16cd19e365f75b3f94ed52b0ce943cddc81ccb"} Jul 09 14:27:48.517123 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.517014 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:48.518680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.518649 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" event={"ID":"41f155d6-09a0-4936-9a7d-8f1458ba9685","Type":"ContainerStarted","Data":"6c53433cd52ce268ca250b7f370834bdb69e32d3d97fac25129b87a6b8323a2e"} Jul 09 14:27:48.518967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.518939 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-tvnvf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 14:27:48.519073 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.518996 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" podUID="f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 14:27:48.599394 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.599350 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" podStartSLOduration=1.7795396860000001 podStartE2EDuration="5.599332494s" totalImagesPullingTime="3.819792808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:27:48.598210538 +0000 UTC m=+432.619293803" watchObservedRunningTime="2026-07-09 14:27:48.599332494 +0000 UTC m=+432.620415757" Jul 09 14:27:48.600297 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.600265 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" podStartSLOduration=2.082730428 podStartE2EDuration="5.600254863s" totalImagesPullingTime="3.517524435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:27:48.560558327 +0000 UTC m=+432.581641590" watchObservedRunningTime="2026-07-09 14:27:48.600254863 +0000 UTC m=+432.621338127" Jul 09 14:27:48.665166 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.665144 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:48.669864 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:48.669841 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:49.523269 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:49.523240 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:27:49.523549 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:49.523529 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tvnvf" Jul 09 14:27:49.524106 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:27:49.524088 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42cjjx" Jul 09 14:28:31.723890 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.723858 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cfd7dc998-pj6gh"] Jul 09 14:28:31.726415 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.726392 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.758079 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.758048 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cfd7dc998-pj6gh"] Jul 09 14:28:31.886689 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.886655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcaebcad-6cca-4fbe-a120-11975ae53423-console-oauth-config\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.886834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.886697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-service-ca\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.886834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.886724 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-oauth-serving-cert\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.886834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.886743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-console-config\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.886951 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.886831 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcaebcad-6cca-4fbe-a120-11975ae53423-console-serving-cert\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.886951 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.886888 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzt2\" (UniqueName: \"kubernetes.io/projected/fcaebcad-6cca-4fbe-a120-11975ae53423-kube-api-access-tlzt2\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.886951 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.886916 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-trusted-ca-bundle\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.988232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.988147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcaebcad-6cca-4fbe-a120-11975ae53423-console-serving-cert\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.988232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.988200 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzt2\" (UniqueName: \"kubernetes.io/projected/fcaebcad-6cca-4fbe-a120-11975ae53423-kube-api-access-tlzt2\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.988232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.988222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-trusted-ca-bundle\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.988442 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.988243 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcaebcad-6cca-4fbe-a120-11975ae53423-console-oauth-config\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.988442 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.988258 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-service-ca\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.988442 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.988376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-oauth-serving-cert\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.988442 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.988412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-console-config\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.989088 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.989061 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-service-ca\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.989215 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.989154 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-console-config\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.989215 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.989156 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-oauth-serving-cert\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.989215 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.989181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcaebcad-6cca-4fbe-a120-11975ae53423-trusted-ca-bundle\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.990688 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.990658 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fcaebcad-6cca-4fbe-a120-11975ae53423-console-oauth-config\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.990828 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.990810 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fcaebcad-6cca-4fbe-a120-11975ae53423-console-serving-cert\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:31.997721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:31.997702 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzt2\" (UniqueName: \"kubernetes.io/projected/fcaebcad-6cca-4fbe-a120-11975ae53423-kube-api-access-tlzt2\") pod \"console-7cfd7dc998-pj6gh\" (UID: \"fcaebcad-6cca-4fbe-a120-11975ae53423\") " pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:32.036214 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:32.036176 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:32.168228 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:32.168202 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cfd7dc998-pj6gh"] Jul 09 14:28:32.170707 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:28:32.170682 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcaebcad_6cca_4fbe_a120_11975ae53423.slice/crio-382dd612d2e74c8acfc9d3206f6fa7293835ee093997884be6710736bac3d827 WatchSource:0}: Error finding container 382dd612d2e74c8acfc9d3206f6fa7293835ee093997884be6710736bac3d827: Status 404 returned error can't find the container with id 382dd612d2e74c8acfc9d3206f6fa7293835ee093997884be6710736bac3d827 Jul 09 14:28:32.717761 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:32.717725 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfd7dc998-pj6gh" event={"ID":"fcaebcad-6cca-4fbe-a120-11975ae53423","Type":"ContainerStarted","Data":"e9bda43334d7d128f311fa6a360bf98e1992a01c43d412080b7e888d8fb125da"} Jul 09 14:28:32.717900 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:32.717786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfd7dc998-pj6gh" event={"ID":"fcaebcad-6cca-4fbe-a120-11975ae53423","Type":"ContainerStarted","Data":"382dd612d2e74c8acfc9d3206f6fa7293835ee093997884be6710736bac3d827"} Jul 09 14:28:32.736924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:32.736878 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cfd7dc998-pj6gh" podStartSLOduration=1.736865267 podStartE2EDuration="1.736865267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:28:32.735715389 +0000 UTC m=+476.756798674" watchObservedRunningTime="2026-07-09 14:28:32.736865267 +0000 UTC m=+476.757948530" Jul 09 14:28:42.036868 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:42.036830 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:42.037309 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:42.036905 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:42.042224 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:42.042202 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:42.766256 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:42.766231 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cfd7dc998-pj6gh" Jul 09 14:28:42.820106 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:42.820075 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77954cdf95-5b59p"] Jul 09 14:28:51.128823 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.128788 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9"] Jul 09 14:28:51.132166 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.132147 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.134880 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.134856 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 14:28:51.135053 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.134861 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 14:28:51.135053 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.134894 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g7ht8\"" Jul 09 14:28:51.141373 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.141349 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9"] Jul 09 14:28:51.144590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.144558 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccp5\" (UniqueName: \"kubernetes.io/projected/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-kube-api-access-mccp5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.144708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.144669 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.144762 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.144736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.245012 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.244986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccp5\" (UniqueName: \"kubernetes.io/projected/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-kube-api-access-mccp5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.245119 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.245023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.245119 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.245054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.245371 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.245351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.245442 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.245393 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.253923 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.253898 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccp5\" (UniqueName: \"kubernetes.io/projected/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-kube-api-access-mccp5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.443353 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.443297 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:51.575186 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.575161 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9"] Jul 09 14:28:51.577409 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:28:51.577379 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30db95bf_dec5_41c5_be6d_d8dfc9fda83f.slice/crio-82da2c2ee1457064b239ac4081940184013b18f4c0089ecaa698e48d6c09f883 WatchSource:0}: Error finding container 82da2c2ee1457064b239ac4081940184013b18f4c0089ecaa698e48d6c09f883: Status 404 returned error can't find the container with id 82da2c2ee1457064b239ac4081940184013b18f4c0089ecaa698e48d6c09f883 Jul 09 14:28:51.725527 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.725478 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55"] Jul 09 14:28:51.728023 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.728009 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.739567 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.739548 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55"] Jul 09 14:28:51.748360 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.748340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.748459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.748366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246js\" (UniqueName: \"kubernetes.io/projected/cb414b74-3776-4cfe-81d6-5fa5b37b6444-kube-api-access-246js\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.748523 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.748500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.808177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.808154 2563 generic.go:356] "Generic (PLEG): container finished" podID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerID="bfd9456774ab27110d065e17866c4444422948b9c95b798b4d43693b6e0b6a29" exitCode=0 Jul 09 14:28:51.808250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.808229 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" event={"ID":"30db95bf-dec5-41c5-be6d-d8dfc9fda83f","Type":"ContainerDied","Data":"bfd9456774ab27110d065e17866c4444422948b9c95b798b4d43693b6e0b6a29"} Jul 09 14:28:51.808295 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.808258 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" event={"ID":"30db95bf-dec5-41c5-be6d-d8dfc9fda83f","Type":"ContainerStarted","Data":"82da2c2ee1457064b239ac4081940184013b18f4c0089ecaa698e48d6c09f883"} Jul 09 14:28:51.848953 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.848932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.849047 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.848959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246js\" (UniqueName: \"kubernetes.io/projected/cb414b74-3776-4cfe-81d6-5fa5b37b6444-kube-api-access-246js\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.849047 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.849003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.849248 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.849231 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.849304 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.849285 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:51.858481 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:51.858455 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-246js\" (UniqueName: \"kubernetes.io/projected/cb414b74-3776-4cfe-81d6-5fa5b37b6444-kube-api-access-246js\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:52.036475 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.036453 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:52.138907 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.137995 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn"] Jul 09 14:28:52.141367 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.141342 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.151547 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.151517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84g8\" (UniqueName: \"kubernetes.io/projected/79a53746-c1b0-449f-bf4d-6551f622556f-kube-api-access-c84g8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.151699 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.151599 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.151699 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.151664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.154078 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.154056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn"] Jul 09 14:28:52.168937 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.168920 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55"] Jul 09 14:28:52.171288 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:28:52.171269 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb414b74_3776_4cfe_81d6_5fa5b37b6444.slice/crio-d1cf518ad82b8d49405c444d3c0463b75516b1b916a4a0d60570445926c9d532 WatchSource:0}: Error finding container d1cf518ad82b8d49405c444d3c0463b75516b1b916a4a0d60570445926c9d532: Status 404 returned error can't find the container with id d1cf518ad82b8d49405c444d3c0463b75516b1b916a4a0d60570445926c9d532 Jul 09 14:28:52.253004 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.252976 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.253004 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.253004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.253133 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.253084 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84g8\" (UniqueName: \"kubernetes.io/projected/79a53746-c1b0-449f-bf4d-6551f622556f-kube-api-access-c84g8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.253405 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.253390 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.253444 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.253403 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.261603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.261580 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84g8\" (UniqueName: \"kubernetes.io/projected/79a53746-c1b0-449f-bf4d-6551f622556f-kube-api-access-c84g8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.452710 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.452682 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:52.535272 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.535243 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh"] Jul 09 14:28:52.538258 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.538216 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.548210 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.548184 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh"] Jul 09 14:28:52.556910 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.556854 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.557000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.556967 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.557060 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.557042 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gs4\" (UniqueName: \"kubernetes.io/projected/6a1edc03-ea74-40c5-b334-8489c611ca8e-kube-api-access-76gs4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.608552 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.608532 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn"] Jul 09 14:28:52.622409 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:28:52.622387 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a53746_c1b0_449f_bf4d_6551f622556f.slice/crio-40df26eb0b3c4355bdf5baec9f646f56be17768fab7fb121455ace32c652ef14 WatchSource:0}: Error finding container 40df26eb0b3c4355bdf5baec9f646f56be17768fab7fb121455ace32c652ef14: Status 404 returned error can't find the container with id 40df26eb0b3c4355bdf5baec9f646f56be17768fab7fb121455ace32c652ef14 Jul 09 14:28:52.658017 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.657996 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.658100 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.658051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gs4\" (UniqueName: \"kubernetes.io/projected/6a1edc03-ea74-40c5-b334-8489c611ca8e-kube-api-access-76gs4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.658145 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.658102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.658395 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.658376 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.658430 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.658400 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.668009 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.667985 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gs4\" (UniqueName: \"kubernetes.io/projected/6a1edc03-ea74-40c5-b334-8489c611ca8e-kube-api-access-76gs4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.814391 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.814333 2563 generic.go:356] "Generic (PLEG): container finished" podID="79a53746-c1b0-449f-bf4d-6551f622556f" containerID="1eb23f40628b773ef2b05b8f0cb18822b50868870d16bb53b3a5f6cb56441780" exitCode=0 Jul 09 14:28:52.814484 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.814426 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" event={"ID":"79a53746-c1b0-449f-bf4d-6551f622556f","Type":"ContainerDied","Data":"1eb23f40628b773ef2b05b8f0cb18822b50868870d16bb53b3a5f6cb56441780"} Jul 09 14:28:52.814484 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.814475 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" event={"ID":"79a53746-c1b0-449f-bf4d-6551f622556f","Type":"ContainerStarted","Data":"40df26eb0b3c4355bdf5baec9f646f56be17768fab7fb121455ace32c652ef14"} Jul 09 14:28:52.822478 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.822454 2563 generic.go:356] "Generic (PLEG): container finished" podID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerID="af4ba31f6925daa11eaed046bea1a6534cfb432f17289a1ad703ad681bbe1d1a" exitCode=0 Jul 09 14:28:52.822564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.822518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" event={"ID":"30db95bf-dec5-41c5-be6d-d8dfc9fda83f","Type":"ContainerDied","Data":"af4ba31f6925daa11eaed046bea1a6534cfb432f17289a1ad703ad681bbe1d1a"} Jul 09 14:28:52.824022 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.823998 2563 generic.go:356] "Generic (PLEG): container finished" podID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerID="59bf23eea16c17b1a4264c5b597ccb780b287865339fa6f9749153afadfbf72f" exitCode=0 Jul 09 14:28:52.824127 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.824022 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" event={"ID":"cb414b74-3776-4cfe-81d6-5fa5b37b6444","Type":"ContainerDied","Data":"59bf23eea16c17b1a4264c5b597ccb780b287865339fa6f9749153afadfbf72f"} Jul 09 14:28:52.824127 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.824065 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" event={"ID":"cb414b74-3776-4cfe-81d6-5fa5b37b6444","Type":"ContainerStarted","Data":"d1cf518ad82b8d49405c444d3c0463b75516b1b916a4a0d60570445926c9d532"} Jul 09 14:28:52.849691 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.849673 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:52.985484 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:52.985461 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh"] Jul 09 14:28:53.015388 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:28:53.015359 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1edc03_ea74_40c5_b334_8489c611ca8e.slice/crio-2aaaa1bf6389568bc85c4a532b0af064cf3c713f365192830a499b39d9efae83 WatchSource:0}: Error finding container 2aaaa1bf6389568bc85c4a532b0af064cf3c713f365192830a499b39d9efae83: Status 404 returned error can't find the container with id 2aaaa1bf6389568bc85c4a532b0af064cf3c713f365192830a499b39d9efae83 Jul 09 14:28:53.832217 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:53.832187 2563 generic.go:356] "Generic (PLEG): container finished" podID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerID="89c8ec78c5a1e029fe0d746d2f3846ea21fcba5aeab9d94552e4695b5e993711" exitCode=0 Jul 09 14:28:53.832639 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:53.832265 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" event={"ID":"30db95bf-dec5-41c5-be6d-d8dfc9fda83f","Type":"ContainerDied","Data":"89c8ec78c5a1e029fe0d746d2f3846ea21fcba5aeab9d94552e4695b5e993711"} Jul 09 14:28:53.833546 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:53.833527 2563 generic.go:356] "Generic (PLEG): container finished" podID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerID="5c00ed4dc5532a27d2af9509cf0c156beb64c06d9655ab133f14843091a0ec73" exitCode=0 Jul 09 14:28:53.833675 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:53.833555 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" event={"ID":"6a1edc03-ea74-40c5-b334-8489c611ca8e","Type":"ContainerDied","Data":"5c00ed4dc5532a27d2af9509cf0c156beb64c06d9655ab133f14843091a0ec73"} Jul 09 14:28:53.833675 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:53.833589 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" event={"ID":"6a1edc03-ea74-40c5-b334-8489c611ca8e","Type":"ContainerStarted","Data":"2aaaa1bf6389568bc85c4a532b0af064cf3c713f365192830a499b39d9efae83"} Jul 09 14:28:54.840693 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:54.840609 2563 generic.go:356] "Generic (PLEG): container finished" podID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerID="fe79cd723e35fd60a05d4646aaea0cf3cc6e96a82848c4f80d1fe91731a2aaf4" exitCode=0 Jul 09 14:28:54.841053 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:54.840689 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" event={"ID":"cb414b74-3776-4cfe-81d6-5fa5b37b6444","Type":"ContainerDied","Data":"fe79cd723e35fd60a05d4646aaea0cf3cc6e96a82848c4f80d1fe91731a2aaf4"} Jul 09 14:28:54.842589 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:54.842567 2563 generic.go:356] "Generic (PLEG): container finished" podID="79a53746-c1b0-449f-bf4d-6551f622556f" containerID="af2972d6b9384d6bd3058685f463d6729d9d59db4b74bed66ff19f1ac250cc21" exitCode=0 Jul 09 14:28:54.842703 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:54.842657 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" event={"ID":"79a53746-c1b0-449f-bf4d-6551f622556f","Type":"ContainerDied","Data":"af2972d6b9384d6bd3058685f463d6729d9d59db4b74bed66ff19f1ac250cc21"} Jul 09 14:28:54.844451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:54.844427 2563 generic.go:356] "Generic (PLEG): container finished" podID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerID="6dbf7439de513bdbcee8d76675d59419128ef36e09eea9d987e5e7940a16d2df" exitCode=0 Jul 09 14:28:54.844535 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:54.844505 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" event={"ID":"6a1edc03-ea74-40c5-b334-8489c611ca8e","Type":"ContainerDied","Data":"6dbf7439de513bdbcee8d76675d59419128ef36e09eea9d987e5e7940a16d2df"} Jul 09 14:28:54.994115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:54.994090 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:55.077057 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.077038 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mccp5\" (UniqueName: \"kubernetes.io/projected/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-kube-api-access-mccp5\") pod \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " Jul 09 14:28:55.077147 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.077103 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-util\") pod \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " Jul 09 14:28:55.077186 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.077160 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-bundle\") pod \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\" (UID: \"30db95bf-dec5-41c5-be6d-d8dfc9fda83f\") " Jul 09 14:28:55.077603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.077584 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-bundle" (OuterVolumeSpecName: "bundle") pod "30db95bf-dec5-41c5-be6d-d8dfc9fda83f" (UID: "30db95bf-dec5-41c5-be6d-d8dfc9fda83f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:55.079005 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.078982 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-kube-api-access-mccp5" (OuterVolumeSpecName: "kube-api-access-mccp5") pod "30db95bf-dec5-41c5-be6d-d8dfc9fda83f" (UID: "30db95bf-dec5-41c5-be6d-d8dfc9fda83f"). InnerVolumeSpecName "kube-api-access-mccp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:28:55.082794 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.082763 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-util" (OuterVolumeSpecName: "util") pod "30db95bf-dec5-41c5-be6d-d8dfc9fda83f" (UID: "30db95bf-dec5-41c5-be6d-d8dfc9fda83f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:55.177967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.177919 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mccp5\" (UniqueName: \"kubernetes.io/projected/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-kube-api-access-mccp5\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:55.177967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.177938 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-util\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:55.177967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.177948 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30db95bf-dec5-41c5-be6d-d8dfc9fda83f-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:55.851836 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.851779 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" Jul 09 14:28:55.852164 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.851771 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9" event={"ID":"30db95bf-dec5-41c5-be6d-d8dfc9fda83f","Type":"ContainerDied","Data":"82da2c2ee1457064b239ac4081940184013b18f4c0089ecaa698e48d6c09f883"} Jul 09 14:28:55.852164 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.851914 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82da2c2ee1457064b239ac4081940184013b18f4c0089ecaa698e48d6c09f883" Jul 09 14:28:55.854095 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.854065 2563 generic.go:356] "Generic (PLEG): container finished" podID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerID="fadc3d6608f5a5dc2d8ade4f20c43a4dac3fc5104eba72c01a8b6065679463e2" exitCode=0 Jul 09 14:28:55.854193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.854151 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" event={"ID":"cb414b74-3776-4cfe-81d6-5fa5b37b6444","Type":"ContainerDied","Data":"fadc3d6608f5a5dc2d8ade4f20c43a4dac3fc5104eba72c01a8b6065679463e2"} Jul 09 14:28:55.856385 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.856359 2563 generic.go:356] "Generic (PLEG): container finished" podID="79a53746-c1b0-449f-bf4d-6551f622556f" containerID="defc211a6917d1da551aa045c5d3f4d60658289c35a77689fb4d2f0611e41959" exitCode=0 Jul 09 14:28:55.856471 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.856448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" event={"ID":"79a53746-c1b0-449f-bf4d-6551f622556f","Type":"ContainerDied","Data":"defc211a6917d1da551aa045c5d3f4d60658289c35a77689fb4d2f0611e41959"} Jul 09 14:28:55.858792 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.858767 2563 generic.go:356] "Generic (PLEG): container finished" podID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerID="6bc605d684fb8948669200079441715200a4e67b3d33508dafc8f70f9ff64338" exitCode=0 Jul 09 14:28:55.858882 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:55.858803 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" event={"ID":"6a1edc03-ea74-40c5-b334-8489c611ca8e","Type":"ContainerDied","Data":"6bc605d684fb8948669200079441715200a4e67b3d33508dafc8f70f9ff64338"} Jul 09 14:28:57.028032 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.027722 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:57.060216 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.060193 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:57.063140 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.063121 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:57.092843 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.092817 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-bundle\") pod \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " Jul 09 14:28:57.092935 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.092881 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-util\") pod \"6a1edc03-ea74-40c5-b334-8489c611ca8e\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " Jul 09 14:28:57.092935 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.092906 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c84g8\" (UniqueName: \"kubernetes.io/projected/79a53746-c1b0-449f-bf4d-6551f622556f-kube-api-access-c84g8\") pod \"79a53746-c1b0-449f-bf4d-6551f622556f\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " Jul 09 14:28:57.093044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.092954 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-246js\" (UniqueName: \"kubernetes.io/projected/cb414b74-3776-4cfe-81d6-5fa5b37b6444-kube-api-access-246js\") pod \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " Jul 09 14:28:57.093044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.092969 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-bundle\") pod \"6a1edc03-ea74-40c5-b334-8489c611ca8e\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " Jul 09 14:28:57.093044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.093023 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-util\") pod \"79a53746-c1b0-449f-bf4d-6551f622556f\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " Jul 09 14:28:57.093044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.093040 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-util\") pod \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\" (UID: \"cb414b74-3776-4cfe-81d6-5fa5b37b6444\") " Jul 09 14:28:57.093246 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.093074 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76gs4\" (UniqueName: \"kubernetes.io/projected/6a1edc03-ea74-40c5-b334-8489c611ca8e-kube-api-access-76gs4\") pod \"6a1edc03-ea74-40c5-b334-8489c611ca8e\" (UID: \"6a1edc03-ea74-40c5-b334-8489c611ca8e\") " Jul 09 14:28:57.093246 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.093101 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-bundle\") pod \"79a53746-c1b0-449f-bf4d-6551f622556f\" (UID: \"79a53746-c1b0-449f-bf4d-6551f622556f\") " Jul 09 14:28:57.093839 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.093576 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-bundle" (OuterVolumeSpecName: "bundle") pod "cb414b74-3776-4cfe-81d6-5fa5b37b6444" (UID: "cb414b74-3776-4cfe-81d6-5fa5b37b6444"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:57.093839 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.093644 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-bundle" (OuterVolumeSpecName: "bundle") pod "6a1edc03-ea74-40c5-b334-8489c611ca8e" (UID: "6a1edc03-ea74-40c5-b334-8489c611ca8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:57.094054 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.094018 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-bundle" (OuterVolumeSpecName: "bundle") pod "79a53746-c1b0-449f-bf4d-6551f622556f" (UID: "79a53746-c1b0-449f-bf4d-6551f622556f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:57.095422 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.095385 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb414b74-3776-4cfe-81d6-5fa5b37b6444-kube-api-access-246js" (OuterVolumeSpecName: "kube-api-access-246js") pod "cb414b74-3776-4cfe-81d6-5fa5b37b6444" (UID: "cb414b74-3776-4cfe-81d6-5fa5b37b6444"). InnerVolumeSpecName "kube-api-access-246js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:28:57.095701 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.095674 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a53746-c1b0-449f-bf4d-6551f622556f-kube-api-access-c84g8" (OuterVolumeSpecName: "kube-api-access-c84g8") pod "79a53746-c1b0-449f-bf4d-6551f622556f" (UID: "79a53746-c1b0-449f-bf4d-6551f622556f"). InnerVolumeSpecName "kube-api-access-c84g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:28:57.096016 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.095996 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1edc03-ea74-40c5-b334-8489c611ca8e-kube-api-access-76gs4" (OuterVolumeSpecName: "kube-api-access-76gs4") pod "6a1edc03-ea74-40c5-b334-8489c611ca8e" (UID: "6a1edc03-ea74-40c5-b334-8489c611ca8e"). InnerVolumeSpecName "kube-api-access-76gs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:28:57.099858 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.099834 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-util" (OuterVolumeSpecName: "util") pod "cb414b74-3776-4cfe-81d6-5fa5b37b6444" (UID: "cb414b74-3776-4cfe-81d6-5fa5b37b6444"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:57.100014 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.099993 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-util" (OuterVolumeSpecName: "util") pod "79a53746-c1b0-449f-bf4d-6551f622556f" (UID: "79a53746-c1b0-449f-bf4d-6551f622556f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:57.100370 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.100347 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-util" (OuterVolumeSpecName: "util") pod "6a1edc03-ea74-40c5-b334-8489c611ca8e" (UID: "6a1edc03-ea74-40c5-b334-8489c611ca8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:28:57.194640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194573 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-util\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194592 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84g8\" (UniqueName: \"kubernetes.io/projected/79a53746-c1b0-449f-bf4d-6551f622556f-kube-api-access-c84g8\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194602 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246js\" (UniqueName: \"kubernetes.io/projected/cb414b74-3776-4cfe-81d6-5fa5b37b6444-kube-api-access-246js\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194612 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1edc03-ea74-40c5-b334-8489c611ca8e-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194638 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-util\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194644 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-util\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194856 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194652 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gs4\" (UniqueName: \"kubernetes.io/projected/6a1edc03-ea74-40c5-b334-8489c611ca8e-kube-api-access-76gs4\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194856 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194660 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a53746-c1b0-449f-bf4d-6551f622556f-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.194856 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.194669 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb414b74-3776-4cfe-81d6-5fa5b37b6444-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:28:57.872597 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.872566 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" Jul 09 14:28:57.872597 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.872576 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55" event={"ID":"cb414b74-3776-4cfe-81d6-5fa5b37b6444","Type":"ContainerDied","Data":"d1cf518ad82b8d49405c444d3c0463b75516b1b916a4a0d60570445926c9d532"} Jul 09 14:28:57.872821 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.872652 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1cf518ad82b8d49405c444d3c0463b75516b1b916a4a0d60570445926c9d532" Jul 09 14:28:57.875235 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.875203 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" Jul 09 14:28:57.875235 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.875228 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn" event={"ID":"79a53746-c1b0-449f-bf4d-6551f622556f","Type":"ContainerDied","Data":"40df26eb0b3c4355bdf5baec9f646f56be17768fab7fb121455ace32c652ef14"} Jul 09 14:28:57.875433 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.875271 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40df26eb0b3c4355bdf5baec9f646f56be17768fab7fb121455ace32c652ef14" Jul 09 14:28:57.877758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.877737 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" Jul 09 14:28:57.877855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.877731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh" event={"ID":"6a1edc03-ea74-40c5-b334-8489c611ca8e","Type":"ContainerDied","Data":"2aaaa1bf6389568bc85c4a532b0af064cf3c713f365192830a499b39d9efae83"} Jul 09 14:28:57.877913 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:28:57.877867 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aaaa1bf6389568bc85c4a532b0af064cf3c713f365192830a499b39d9efae83" Jul 09 14:29:07.844300 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:07.844242 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77954cdf95-5b59p" podUID="33ff4139-ffc9-45ca-9f48-9ab908365df3" containerName="console" containerID="cri-o://0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75" gracePeriod=15 Jul 09 14:29:08.079126 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.079105 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:29:08.174509 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.174450 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-config\") pod \"33ff4139-ffc9-45ca-9f48-9ab908365df3\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " Jul 09 14:29:08.174509 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.174479 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-trusted-ca-bundle\") pod \"33ff4139-ffc9-45ca-9f48-9ab908365df3\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " Jul 09 14:29:08.174509 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.174503 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zgrs\" (UniqueName: \"kubernetes.io/projected/33ff4139-ffc9-45ca-9f48-9ab908365df3-kube-api-access-9zgrs\") pod \"33ff4139-ffc9-45ca-9f48-9ab908365df3\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " Jul 09 14:29:08.174770 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.174517 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-service-ca\") pod \"33ff4139-ffc9-45ca-9f48-9ab908365df3\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " Jul 09 14:29:08.174770 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.174550 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-serving-cert\") pod \"33ff4139-ffc9-45ca-9f48-9ab908365df3\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " Jul 09 14:29:08.174770 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.174592 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-oauth-config\") pod \"33ff4139-ffc9-45ca-9f48-9ab908365df3\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " Jul 09 14:29:08.174770 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.174657 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-oauth-serving-cert\") pod \"33ff4139-ffc9-45ca-9f48-9ab908365df3\" (UID: \"33ff4139-ffc9-45ca-9f48-9ab908365df3\") " Jul 09 14:29:08.175180 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.175156 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "33ff4139-ffc9-45ca-9f48-9ab908365df3" (UID: "33ff4139-ffc9-45ca-9f48-9ab908365df3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:29:08.175326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.175154 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-config" (OuterVolumeSpecName: "console-config") pod "33ff4139-ffc9-45ca-9f48-9ab908365df3" (UID: "33ff4139-ffc9-45ca-9f48-9ab908365df3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:29:08.175395 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.175185 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-service-ca" (OuterVolumeSpecName: "service-ca") pod "33ff4139-ffc9-45ca-9f48-9ab908365df3" (UID: "33ff4139-ffc9-45ca-9f48-9ab908365df3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:29:08.175460 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.175325 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "33ff4139-ffc9-45ca-9f48-9ab908365df3" (UID: "33ff4139-ffc9-45ca-9f48-9ab908365df3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:29:08.176816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.176788 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "33ff4139-ffc9-45ca-9f48-9ab908365df3" (UID: "33ff4139-ffc9-45ca-9f48-9ab908365df3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:29:08.177118 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.177096 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ff4139-ffc9-45ca-9f48-9ab908365df3-kube-api-access-9zgrs" (OuterVolumeSpecName: "kube-api-access-9zgrs") pod "33ff4139-ffc9-45ca-9f48-9ab908365df3" (UID: "33ff4139-ffc9-45ca-9f48-9ab908365df3"). InnerVolumeSpecName "kube-api-access-9zgrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:29:08.177250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.177223 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "33ff4139-ffc9-45ca-9f48-9ab908365df3" (UID: "33ff4139-ffc9-45ca-9f48-9ab908365df3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:29:08.275675 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.275650 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:29:08.275675 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.275669 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-oauth-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:29:08.275675 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.275679 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-oauth-serving-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:29:08.275834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.275688 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-console-config\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:29:08.275834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.275696 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-trusted-ca-bundle\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:29:08.275834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.275705 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgrs\" (UniqueName: \"kubernetes.io/projected/33ff4139-ffc9-45ca-9f48-9ab908365df3-kube-api-access-9zgrs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:29:08.275834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.275714 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33ff4139-ffc9-45ca-9f48-9ab908365df3-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:29:08.945573 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.945485 2563 generic.go:356] "Generic (PLEG): container finished" podID="33ff4139-ffc9-45ca-9f48-9ab908365df3" containerID="0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75" exitCode=0 Jul 09 14:29:08.945573 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.945536 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77954cdf95-5b59p" Jul 09 14:29:08.946056 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.945584 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77954cdf95-5b59p" event={"ID":"33ff4139-ffc9-45ca-9f48-9ab908365df3","Type":"ContainerDied","Data":"0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75"} Jul 09 14:29:08.946056 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.945679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77954cdf95-5b59p" event={"ID":"33ff4139-ffc9-45ca-9f48-9ab908365df3","Type":"ContainerDied","Data":"ebc2a12a5a046cdcd39bbc5a1f2c27815e1950578aad3ddc67c942451142a1e6"} Jul 09 14:29:08.946056 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.945718 2563 scope.go:117] "RemoveContainer" containerID="0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75" Jul 09 14:29:08.955868 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.955847 2563 scope.go:117] "RemoveContainer" containerID="0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75" Jul 09 14:29:08.956163 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:29:08.956141 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75\": container with ID starting with 0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75 not found: ID does not exist" containerID="0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75" Jul 09 14:29:08.956226 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.956170 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75"} err="failed to get container status \"0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75\": rpc error: code = NotFound desc = could not find container \"0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75\": container with ID starting with 0ccad73f8e88f36c0934f33896d1548e0945794a55cf19eb4715514089223c75 not found: ID does not exist" Jul 09 14:29:08.974748 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.974720 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77954cdf95-5b59p"] Jul 09 14:29:08.984125 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:08.984095 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77954cdf95-5b59p"] Jul 09 14:29:10.582675 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:29:10.582609 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ff4139-ffc9-45ca-9f48-9ab908365df3" path="/var/lib/kubelet/pods/33ff4139-ffc9-45ca-9f48-9ab908365df3/volumes" Jul 09 14:30:36.443695 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:30:36.443667 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:30:36.450341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:30:36.450321 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:30:36.450996 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:30:36.450977 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:30:36.457079 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:30:36.457061 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:31:00.009684 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009592 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8487b45dc4-v96zg"] Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009917 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerName="util" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009929 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerName="util" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009939 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009945 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009959 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a53746-c1b0-449f-bf4d-6551f622556f" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009965 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a53746-c1b0-449f-bf4d-6551f622556f" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009976 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009981 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009988 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ff4139-ffc9-45ca-9f48-9ab908365df3" containerName="console" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009993 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ff4139-ffc9-45ca-9f48-9ab908365df3" containerName="console" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.009999 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010005 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010011 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a53746-c1b0-449f-bf4d-6551f622556f" containerName="util" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010017 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a53746-c1b0-449f-bf4d-6551f622556f" containerName="util" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010024 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a53746-c1b0-449f-bf4d-6551f622556f" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010031 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a53746-c1b0-449f-bf4d-6551f622556f" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010039 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerName="util" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010044 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerName="util" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010050 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010054 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerName="extract" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010066 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010072 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010079 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010083 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerName="pull" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010090 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerName="util" Jul 09 14:31:00.010115 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010095 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerName="util" Jul 09 14:31:00.011259 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010146 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="30db95bf-dec5-41c5-be6d-d8dfc9fda83f" containerName="extract" Jul 09 14:31:00.011259 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010155 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ff4139-ffc9-45ca-9f48-9ab908365df3" containerName="console" Jul 09 14:31:00.011259 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010162 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1edc03-ea74-40c5-b334-8489c611ca8e" containerName="extract" Jul 09 14:31:00.011259 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010167 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb414b74-3776-4cfe-81d6-5fa5b37b6444" containerName="extract" Jul 09 14:31:00.011259 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.010174 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a53746-c1b0-449f-bf4d-6551f622556f" containerName="extract" Jul 09 14:31:00.013160 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.013144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.015736 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.015711 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 14:31:00.015736 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.015710 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 14:31:00.015908 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.015785 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vjpkb\"" Jul 09 14:31:00.025962 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.025944 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8487b45dc4-v96zg"] Jul 09 14:31:00.116263 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.116238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert\") pod \"maas-controller-8487b45dc4-v96zg\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.116389 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.116274 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgz5\" (UniqueName: \"kubernetes.io/projected/4cfb223e-9383-4722-9556-b3ea407dfcc7-kube-api-access-4lgz5\") pod \"maas-controller-8487b45dc4-v96zg\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.216881 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.216854 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert\") pod \"maas-controller-8487b45dc4-v96zg\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.216987 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.216885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgz5\" (UniqueName: \"kubernetes.io/projected/4cfb223e-9383-4722-9556-b3ea407dfcc7-kube-api-access-4lgz5\") pod \"maas-controller-8487b45dc4-v96zg\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.217047 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:31:00.216990 2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 14:31:00.217047 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:31:00.217044 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert podName:4cfb223e-9383-4722-9556-b3ea407dfcc7 nodeName:}" failed. No retries permitted until 2026-07-09 14:31:00.717029447 +0000 UTC m=+624.738112693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert") pod "maas-controller-8487b45dc4-v96zg" (UID: "4cfb223e-9383-4722-9556-b3ea407dfcc7") : secret "maas-controller-webhook-cert" not found Jul 09 14:31:00.226582 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.226562 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dff748cfb-9dg78"] Jul 09 14:31:00.229674 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.229659 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.238410 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.238387 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgz5\" (UniqueName: \"kubernetes.io/projected/4cfb223e-9383-4722-9556-b3ea407dfcc7-kube-api-access-4lgz5\") pod \"maas-controller-8487b45dc4-v96zg\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.246959 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.246941 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dff748cfb-9dg78"] Jul 09 14:31:00.317721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.317672 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8lll\" (UniqueName: \"kubernetes.io/projected/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-kube-api-access-s8lll\") pod \"maas-controller-7dff748cfb-9dg78\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.317721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.317700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-webhook-cert\") pod \"maas-controller-7dff748cfb-9dg78\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.418488 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.418462 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8lll\" (UniqueName: \"kubernetes.io/projected/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-kube-api-access-s8lll\") pod \"maas-controller-7dff748cfb-9dg78\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.418573 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.418494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-webhook-cert\") pod \"maas-controller-7dff748cfb-9dg78\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.420889 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.420867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-webhook-cert\") pod \"maas-controller-7dff748cfb-9dg78\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.428465 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.428443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8lll\" (UniqueName: \"kubernetes.io/projected/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-kube-api-access-s8lll\") pod \"maas-controller-7dff748cfb-9dg78\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.538635 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.538596 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:00.678242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.678211 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dff748cfb-9dg78"] Jul 09 14:31:00.679534 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:31:00.679508 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b8d548_18de_4dac_b81d_1cf0bd6740d3.slice/crio-77fe8bda99479704fe8f715e27880e3010e6f750dd99107ba4db0df7971aab4b WatchSource:0}: Error finding container 77fe8bda99479704fe8f715e27880e3010e6f750dd99107ba4db0df7971aab4b: Status 404 returned error can't find the container with id 77fe8bda99479704fe8f715e27880e3010e6f750dd99107ba4db0df7971aab4b Jul 09 14:31:00.721196 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.721174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert\") pod \"maas-controller-8487b45dc4-v96zg\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.723355 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.723338 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert\") pod \"maas-controller-8487b45dc4-v96zg\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:00.922725 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:00.922645 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:01.057150 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:01.057124 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8487b45dc4-v96zg"] Jul 09 14:31:01.059013 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:31:01.058987 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfb223e_9383_4722_9556_b3ea407dfcc7.slice/crio-84a8f30592d7d0d6980b040ef1e6e9f65b94e22881d76ea0f2f09fabbe79fce8 WatchSource:0}: Error finding container 84a8f30592d7d0d6980b040ef1e6e9f65b94e22881d76ea0f2f09fabbe79fce8: Status 404 returned error can't find the container with id 84a8f30592d7d0d6980b040ef1e6e9f65b94e22881d76ea0f2f09fabbe79fce8 Jul 09 14:31:01.527206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:01.527166 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8487b45dc4-v96zg" event={"ID":"4cfb223e-9383-4722-9556-b3ea407dfcc7","Type":"ContainerStarted","Data":"84a8f30592d7d0d6980b040ef1e6e9f65b94e22881d76ea0f2f09fabbe79fce8"} Jul 09 14:31:01.528690 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:01.528662 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dff748cfb-9dg78" event={"ID":"e0b8d548-18de-4dac-b81d-1cf0bd6740d3","Type":"ContainerStarted","Data":"77fe8bda99479704fe8f715e27880e3010e6f750dd99107ba4db0df7971aab4b"} Jul 09 14:31:03.541688 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:03.541649 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8487b45dc4-v96zg" event={"ID":"4cfb223e-9383-4722-9556-b3ea407dfcc7","Type":"ContainerStarted","Data":"360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d"} Jul 09 14:31:03.542168 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:03.541717 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:03.543169 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:03.543136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dff748cfb-9dg78" event={"ID":"e0b8d548-18de-4dac-b81d-1cf0bd6740d3","Type":"ContainerStarted","Data":"f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7"} Jul 09 14:31:03.543253 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:03.543196 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:03.568002 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:03.567957 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8487b45dc4-v96zg" podStartSLOduration=2.472055128 podStartE2EDuration="4.567943489s" totalImagesPullingTime="2.095888361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:31:03.565744393 +0000 UTC m=+627.586827658" watchObservedRunningTime="2026-07-09 14:31:03.567943489 +0000 UTC m=+627.589026752" Jul 09 14:31:03.592514 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:03.592472 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dff748cfb-9dg78" podStartSLOduration=1.106292717 podStartE2EDuration="3.592459549s" totalImagesPullingTime="2.486166832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:31:03.591580704 +0000 UTC m=+627.612663988" watchObservedRunningTime="2026-07-09 14:31:03.592459549 +0000 UTC m=+627.613542825" Jul 09 14:31:14.554423 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:14.554384 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:31:14.554816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:14.554674 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:14.644355 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:14.644322 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8487b45dc4-v96zg"] Jul 09 14:31:14.644585 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:14.644557 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8487b45dc4-v96zg" podUID="4cfb223e-9383-4722-9556-b3ea407dfcc7" containerName="manager" containerID="cri-o://360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d" gracePeriod=30 Jul 09 14:31:14.889232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:14.889211 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:15.032578 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.032552 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert\") pod \"4cfb223e-9383-4722-9556-b3ea407dfcc7\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " Jul 09 14:31:15.032754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.032628 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lgz5\" (UniqueName: \"kubernetes.io/projected/4cfb223e-9383-4722-9556-b3ea407dfcc7-kube-api-access-4lgz5\") pod \"4cfb223e-9383-4722-9556-b3ea407dfcc7\" (UID: \"4cfb223e-9383-4722-9556-b3ea407dfcc7\") " Jul 09 14:31:15.034634 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.034586 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfb223e-9383-4722-9556-b3ea407dfcc7-kube-api-access-4lgz5" (OuterVolumeSpecName: "kube-api-access-4lgz5") pod "4cfb223e-9383-4722-9556-b3ea407dfcc7" (UID: "4cfb223e-9383-4722-9556-b3ea407dfcc7"). InnerVolumeSpecName "kube-api-access-4lgz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:31:15.034634 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.034610 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4cfb223e-9383-4722-9556-b3ea407dfcc7" (UID: "4cfb223e-9383-4722-9556-b3ea407dfcc7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:31:15.133215 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.133138 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4cfb223e-9383-4722-9556-b3ea407dfcc7-webhook-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:31:15.133215 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.133165 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lgz5\" (UniqueName: \"kubernetes.io/projected/4cfb223e-9383-4722-9556-b3ea407dfcc7-kube-api-access-4lgz5\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:31:15.607270 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.607236 2563 generic.go:356] "Generic (PLEG): container finished" podID="4cfb223e-9383-4722-9556-b3ea407dfcc7" containerID="360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d" exitCode=0 Jul 09 14:31:15.607697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.607276 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8487b45dc4-v96zg" Jul 09 14:31:15.607697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.607324 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8487b45dc4-v96zg" event={"ID":"4cfb223e-9383-4722-9556-b3ea407dfcc7","Type":"ContainerDied","Data":"360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d"} Jul 09 14:31:15.607697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.607373 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8487b45dc4-v96zg" event={"ID":"4cfb223e-9383-4722-9556-b3ea407dfcc7","Type":"ContainerDied","Data":"84a8f30592d7d0d6980b040ef1e6e9f65b94e22881d76ea0f2f09fabbe79fce8"} Jul 09 14:31:15.607697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.607398 2563 scope.go:117] "RemoveContainer" containerID="360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d" Jul 09 14:31:15.616129 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.616112 2563 scope.go:117] "RemoveContainer" containerID="360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d" Jul 09 14:31:15.616383 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:31:15.616364 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d\": container with ID starting with 360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d not found: ID does not exist" containerID="360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d" Jul 09 14:31:15.616430 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.616391 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d"} err="failed to get container status \"360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d\": rpc error: code = NotFound desc = could not find container \"360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d\": container with ID starting with 360e058246d178f723f94a37403bfd9c2725645df07367b6b036830c4830944d not found: ID does not exist" Jul 09 14:31:15.631196 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.631178 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8487b45dc4-v96zg"] Jul 09 14:31:15.638867 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:15.638847 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8487b45dc4-v96zg"] Jul 09 14:31:16.575341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:16.575310 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cfb223e-9383-4722-9556-b3ea407dfcc7" path="/var/lib/kubelet/pods/4cfb223e-9383-4722-9556-b3ea407dfcc7/volumes" Jul 09 14:31:35.316214 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.316178 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h"] Jul 09 14:31:35.316710 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.316500 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cfb223e-9383-4722-9556-b3ea407dfcc7" containerName="manager" Jul 09 14:31:35.316710 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.316511 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cfb223e-9383-4722-9556-b3ea407dfcc7" containerName="manager" Jul 09 14:31:35.316710 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.316576 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cfb223e-9383-4722-9556-b3ea407dfcc7" containerName="manager" Jul 09 14:31:35.319484 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.319467 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.325865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.325775 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 14:31:35.326043 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.326020 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 14:31:35.326128 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.326110 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 14:31:35.336913 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.336888 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h"] Jul 09 14:31:35.377447 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.377414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e479426-d010-4701-ad5a-1bda78affe3e-maas-api-tls\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.377652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.377456 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdft\" (UniqueName: \"kubernetes.io/projected/7e479426-d010-4701-ad5a-1bda78affe3e-kube-api-access-lsdft\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.377652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.377522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e479426-d010-4701-ad5a-1bda78affe3e-openshift-service-ca\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.478170 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.478123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e479426-d010-4701-ad5a-1bda78affe3e-openshift-service-ca\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.478336 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.478252 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e479426-d010-4701-ad5a-1bda78affe3e-maas-api-tls\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.478336 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.478291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdft\" (UniqueName: \"kubernetes.io/projected/7e479426-d010-4701-ad5a-1bda78affe3e-kube-api-access-lsdft\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.478437 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:31:35.478414 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 09 14:31:35.478501 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:31:35.478490 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e479426-d010-4701-ad5a-1bda78affe3e-maas-api-tls podName:7e479426-d010-4701-ad5a-1bda78affe3e nodeName:}" failed. No retries permitted until 2026-07-09 14:31:35.978468958 +0000 UTC m=+659.999552200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e479426-d010-4701-ad5a-1bda78affe3e-maas-api-tls") pod "maas-api-57b5dcc795-n2c4h" (UID: "7e479426-d010-4701-ad5a-1bda78affe3e") : secret "maas-api-serving-cert" not found Jul 09 14:31:35.478903 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.478880 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e479426-d010-4701-ad5a-1bda78affe3e-openshift-service-ca\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.489140 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.489114 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdft\" (UniqueName: \"kubernetes.io/projected/7e479426-d010-4701-ad5a-1bda78affe3e-kube-api-access-lsdft\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.982539 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.982502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e479426-d010-4701-ad5a-1bda78affe3e-maas-api-tls\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:35.985011 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:35.984983 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e479426-d010-4701-ad5a-1bda78affe3e-maas-api-tls\") pod \"maas-api-57b5dcc795-n2c4h\" (UID: \"7e479426-d010-4701-ad5a-1bda78affe3e\") " pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:36.229416 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:36.229382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:36.373998 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:36.373971 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h"] Jul 09 14:31:36.376060 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:31:36.376030 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e479426_d010_4701_ad5a_1bda78affe3e.slice/crio-151a96259907ca7fb0983d480118600f920074cf71012874a36aa6fdd99707c2 WatchSource:0}: Error finding container 151a96259907ca7fb0983d480118600f920074cf71012874a36aa6fdd99707c2: Status 404 returned error can't find the container with id 151a96259907ca7fb0983d480118600f920074cf71012874a36aa6fdd99707c2 Jul 09 14:31:36.377340 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:36.377318 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:31:36.713944 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:36.713905 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" event={"ID":"7e479426-d010-4701-ad5a-1bda78affe3e","Type":"ContainerStarted","Data":"151a96259907ca7fb0983d480118600f920074cf71012874a36aa6fdd99707c2"} Jul 09 14:31:38.725492 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:38.725397 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" event={"ID":"7e479426-d010-4701-ad5a-1bda78affe3e","Type":"ContainerStarted","Data":"75897f728a380e1559ec96e78039311baf5336933b41988afd9c456f9fa0c44d"} Jul 09 14:31:38.725893 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:38.725564 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:38.757160 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:38.757106 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" podStartSLOduration=1.701393616 podStartE2EDuration="3.757088529s" totalImagesPullingTime="2.055694913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:31:38.755370415 +0000 UTC m=+662.776453678" watchObservedRunningTime="2026-07-09 14:31:38.757088529 +0000 UTC m=+662.778171841" Jul 09 14:31:44.735965 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:44.735939 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57b5dcc795-n2c4h" Jul 09 14:31:58.315461 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.315423 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp"] Jul 09 14:31:58.318190 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.318170 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.321901 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.321875 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 14:31:58.322037 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.321975 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ndfhx\"" Jul 09 14:31:58.322037 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.321997 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 14:31:58.322037 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.322023 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 14:31:58.329701 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.329675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp"] Jul 09 14:31:58.342606 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.342580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krpb\" (UniqueName: \"kubernetes.io/projected/1f5b66c3-ee99-4958-a63e-26d3941de8fa-kube-api-access-4krpb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.342753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.342611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.342753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.342680 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.342753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.342714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.342884 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.342771 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5b66c3-ee99-4958-a63e-26d3941de8fa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.342884 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.342807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.443963 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.443928 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krpb\" (UniqueName: \"kubernetes.io/projected/1f5b66c3-ee99-4958-a63e-26d3941de8fa-kube-api-access-4krpb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.443963 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.443965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.444244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.443991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.444244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.444021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.444244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.444069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5b66c3-ee99-4958-a63e-26d3941de8fa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.444244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.444097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.444468 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.444444 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.444517 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.444465 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.444555 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.444542 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.446203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.446181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5b66c3-ee99-4958-a63e-26d3941de8fa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.446481 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.446463 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5b66c3-ee99-4958-a63e-26d3941de8fa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.453495 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.453468 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krpb\" (UniqueName: \"kubernetes.io/projected/1f5b66c3-ee99-4958-a63e-26d3941de8fa-kube-api-access-4krpb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp\" (UID: \"1f5b66c3-ee99-4958-a63e-26d3941de8fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.629292 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.629213 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:31:58.762274 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.762249 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp"] Jul 09 14:31:58.765129 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:31:58.765086 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5b66c3_ee99_4958_a63e_26d3941de8fa.slice/crio-585cfbe22a592b4f58c143e04c2f250da336a2bc45776e3e4b07dbb6f2b4705a WatchSource:0}: Error finding container 585cfbe22a592b4f58c143e04c2f250da336a2bc45776e3e4b07dbb6f2b4705a: Status 404 returned error can't find the container with id 585cfbe22a592b4f58c143e04c2f250da336a2bc45776e3e4b07dbb6f2b4705a Jul 09 14:31:58.824873 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:31:58.824834 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" event={"ID":"1f5b66c3-ee99-4958-a63e-26d3941de8fa","Type":"ContainerStarted","Data":"585cfbe22a592b4f58c143e04c2f250da336a2bc45776e3e4b07dbb6f2b4705a"} Jul 09 14:32:00.836813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:32:00.836779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" event={"ID":"1f5b66c3-ee99-4958-a63e-26d3941de8fa","Type":"ContainerStarted","Data":"8684a0984e910c379fa05abefaa7bee06b173796473a8e50fe6b1f645566972d"} Jul 09 14:32:00.857004 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:32:00.856962 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" podStartSLOduration=1.756817319 podStartE2EDuration="2.856946533s" totalImagesPullingTime="1.100129214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:32:00.855960925 +0000 UTC m=+684.877044208" watchObservedRunningTime="2026-07-09 14:32:00.856946533 +0000 UTC m=+684.878029797" Jul 09 14:32:08.629970 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:32:08.629932 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:32:08.630576 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:32:08.630009 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:32:08.642084 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:32:08.642056 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:32:08.887377 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:32:08.887285 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp" Jul 09 14:35:05.238598 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.238564 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dff748cfb-9dg78"] Jul 09 14:35:05.239132 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.238773 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7dff748cfb-9dg78" podUID="e0b8d548-18de-4dac-b81d-1cf0bd6740d3" containerName="manager" containerID="cri-o://f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7" gracePeriod=30 Jul 09 14:35:05.479332 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.479307 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:35:05.554514 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.554484 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8lll\" (UniqueName: \"kubernetes.io/projected/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-kube-api-access-s8lll\") pod \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " Jul 09 14:35:05.554681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.554520 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-webhook-cert\") pod \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\" (UID: \"e0b8d548-18de-4dac-b81d-1cf0bd6740d3\") " Jul 09 14:35:05.556409 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.556379 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-kube-api-access-s8lll" (OuterVolumeSpecName: "kube-api-access-s8lll") pod "e0b8d548-18de-4dac-b81d-1cf0bd6740d3" (UID: "e0b8d548-18de-4dac-b81d-1cf0bd6740d3"). InnerVolumeSpecName "kube-api-access-s8lll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:35:05.556409 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.556393 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e0b8d548-18de-4dac-b81d-1cf0bd6740d3" (UID: "e0b8d548-18de-4dac-b81d-1cf0bd6740d3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:35:05.655469 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.655440 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8lll\" (UniqueName: \"kubernetes.io/projected/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-kube-api-access-s8lll\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:35:05.655469 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.655464 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0b8d548-18de-4dac-b81d-1cf0bd6740d3-webhook-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:35:05.766597 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.766564 2563 generic.go:356] "Generic (PLEG): container finished" podID="e0b8d548-18de-4dac-b81d-1cf0bd6740d3" containerID="f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7" exitCode=0 Jul 09 14:35:05.766699 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.766607 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dff748cfb-9dg78" Jul 09 14:35:05.766699 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.766648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dff748cfb-9dg78" event={"ID":"e0b8d548-18de-4dac-b81d-1cf0bd6740d3","Type":"ContainerDied","Data":"f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7"} Jul 09 14:35:05.766797 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.766701 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dff748cfb-9dg78" event={"ID":"e0b8d548-18de-4dac-b81d-1cf0bd6740d3","Type":"ContainerDied","Data":"77fe8bda99479704fe8f715e27880e3010e6f750dd99107ba4db0df7971aab4b"} Jul 09 14:35:05.766797 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.766722 2563 scope.go:117] "RemoveContainer" containerID="f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7" Jul 09 14:35:05.775370 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.775354 2563 scope.go:117] "RemoveContainer" containerID="f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7" Jul 09 14:35:05.775650 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:35:05.775632 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7\": container with ID starting with f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7 not found: ID does not exist" containerID="f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7" Jul 09 14:35:05.775707 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.775660 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7"} err="failed to get container status \"f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7\": rpc error: code = NotFound desc = could not find container \"f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7\": container with ID starting with f1f6ab732e7b95cfd1fa8175678cac87154f319ebc34ec54ac1fe0755c22edf7 not found: ID does not exist" Jul 09 14:35:05.787254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.787230 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dff748cfb-9dg78"] Jul 09 14:35:05.791111 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:05.791089 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dff748cfb-9dg78"] Jul 09 14:35:06.553309 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.553278 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dff748cfb-tfmfr"] Jul 09 14:35:06.553672 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.553594 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b8d548-18de-4dac-b81d-1cf0bd6740d3" containerName="manager" Jul 09 14:35:06.553672 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.553606 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b8d548-18de-4dac-b81d-1cf0bd6740d3" containerName="manager" Jul 09 14:35:06.553754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.553695 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0b8d548-18de-4dac-b81d-1cf0bd6740d3" containerName="manager" Jul 09 14:35:06.556193 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.556176 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.559533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.559509 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 14:35:06.559661 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.559537 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 14:35:06.559661 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.559538 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vjpkb\"" Jul 09 14:35:06.567418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.567396 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dff748cfb-tfmfr"] Jul 09 14:35:06.574454 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.574433 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0b8d548-18de-4dac-b81d-1cf0bd6740d3" path="/var/lib/kubelet/pods/e0b8d548-18de-4dac-b81d-1cf0bd6740d3/volumes" Jul 09 14:35:06.663598 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.663576 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngd5w\" (UniqueName: \"kubernetes.io/projected/ada34cf0-f30d-4630-a359-64fb51d51f4d-kube-api-access-ngd5w\") pod \"maas-controller-7dff748cfb-tfmfr\" (UID: \"ada34cf0-f30d-4630-a359-64fb51d51f4d\") " pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.663704 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.663607 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ada34cf0-f30d-4630-a359-64fb51d51f4d-webhook-cert\") pod \"maas-controller-7dff748cfb-tfmfr\" (UID: \"ada34cf0-f30d-4630-a359-64fb51d51f4d\") " pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.764962 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.764932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngd5w\" (UniqueName: \"kubernetes.io/projected/ada34cf0-f30d-4630-a359-64fb51d51f4d-kube-api-access-ngd5w\") pod \"maas-controller-7dff748cfb-tfmfr\" (UID: \"ada34cf0-f30d-4630-a359-64fb51d51f4d\") " pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.765065 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.764972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ada34cf0-f30d-4630-a359-64fb51d51f4d-webhook-cert\") pod \"maas-controller-7dff748cfb-tfmfr\" (UID: \"ada34cf0-f30d-4630-a359-64fb51d51f4d\") " pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.767221 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.767201 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ada34cf0-f30d-4630-a359-64fb51d51f4d-webhook-cert\") pod \"maas-controller-7dff748cfb-tfmfr\" (UID: \"ada34cf0-f30d-4630-a359-64fb51d51f4d\") " pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.773599 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.773580 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngd5w\" (UniqueName: \"kubernetes.io/projected/ada34cf0-f30d-4630-a359-64fb51d51f4d-kube-api-access-ngd5w\") pod \"maas-controller-7dff748cfb-tfmfr\" (UID: \"ada34cf0-f30d-4630-a359-64fb51d51f4d\") " pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.865841 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.865784 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:06.993685 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:06.993648 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dff748cfb-tfmfr"] Jul 09 14:35:06.996461 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:35:06.996434 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada34cf0_f30d_4630_a359_64fb51d51f4d.slice/crio-9964698a0a485d8ba30ac40c1010b98f82e61b8ef6022764db6172e0486850dc WatchSource:0}: Error finding container 9964698a0a485d8ba30ac40c1010b98f82e61b8ef6022764db6172e0486850dc: Status 404 returned error can't find the container with id 9964698a0a485d8ba30ac40c1010b98f82e61b8ef6022764db6172e0486850dc Jul 09 14:35:07.779428 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:07.779395 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dff748cfb-tfmfr" event={"ID":"ada34cf0-f30d-4630-a359-64fb51d51f4d","Type":"ContainerStarted","Data":"9694b1eb723a8760b6c43cc4d3c32b8621ec659345f4d4c06e13a16e1540a563"} Jul 09 14:35:07.779792 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:07.779521 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:07.779792 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:07.779543 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dff748cfb-tfmfr" event={"ID":"ada34cf0-f30d-4630-a359-64fb51d51f4d","Type":"ContainerStarted","Data":"9964698a0a485d8ba30ac40c1010b98f82e61b8ef6022764db6172e0486850dc"} Jul 09 14:35:07.796920 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:07.796880 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7dff748cfb-tfmfr" podStartSLOduration=1.381519117 podStartE2EDuration="1.796865255s" totalImagesPullingTime="415.346138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:35:07.795653509 +0000 UTC m=+871.816736803" watchObservedRunningTime="2026-07-09 14:35:07.796865255 +0000 UTC m=+871.817948529" Jul 09 14:35:18.790519 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:18.790491 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dff748cfb-tfmfr" Jul 09 14:35:36.480589 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:36.480508 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:35:36.485377 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:36.485355 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:35:36.487566 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:36.487547 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:35:36.492307 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:35:36.492291 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:40:36.518225 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:40:36.518192 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:40:36.520799 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:40:36.520778 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:40:36.525040 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:40:36.525021 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:40:36.527127 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:40:36.527107 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:45:00.157205 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.157168 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7"] Jul 09 14:45:00.160438 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.160422 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" Jul 09 14:45:00.163019 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.162998 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:45:00.186486 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.186461 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7"] Jul 09 14:45:00.256377 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.256344 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhbb\" (UniqueName: \"kubernetes.io/projected/72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9-kube-api-access-rlhbb\") pod \"maas-api-key-cleanup-29726805-ccdz7\" (UID: \"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" Jul 09 14:45:00.357242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.357204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhbb\" (UniqueName: \"kubernetes.io/projected/72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9-kube-api-access-rlhbb\") pod \"maas-api-key-cleanup-29726805-ccdz7\" (UID: \"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" Jul 09 14:45:00.366297 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.366277 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhbb\" (UniqueName: \"kubernetes.io/projected/72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9-kube-api-access-rlhbb\") pod \"maas-api-key-cleanup-29726805-ccdz7\" (UID: \"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" Jul 09 14:45:00.470578 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.470472 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" Jul 09 14:45:00.603306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.603274 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7"] Jul 09 14:45:00.605978 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:45:00.605949 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72eaa91a_d8f2_4c5f_88b5_6237b1ebbdb9.slice/crio-e08a3fcfba2de4c096f192ceecbf8ecc8c863c29e2e8522b92facd6ac2ce6255 WatchSource:0}: Error finding container e08a3fcfba2de4c096f192ceecbf8ecc8c863c29e2e8522b92facd6ac2ce6255: Status 404 returned error can't find the container with id e08a3fcfba2de4c096f192ceecbf8ecc8c863c29e2e8522b92facd6ac2ce6255 Jul 09 14:45:00.607756 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.607737 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:45:00.827476 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:00.827439 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" event={"ID":"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9","Type":"ContainerStarted","Data":"e08a3fcfba2de4c096f192ceecbf8ecc8c863c29e2e8522b92facd6ac2ce6255"} Jul 09 14:45:05.856307 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:05.856274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" event={"ID":"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9","Type":"ContainerStarted","Data":"979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3"} Jul 09 14:45:05.872981 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:05.872936 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" podStartSLOduration=1.613720674 podStartE2EDuration="5.872921639s" totalImagesPullingTime="4.259200965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:45:05.871397822 +0000 UTC m=+1469.892481107" watchObservedRunningTime="2026-07-09 14:45:05.872921639 +0000 UTC m=+1469.894004941" Jul 09 14:45:36.560296 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:36.560264 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:45:36.560856 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:36.560591 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:45:36.567191 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:36.567171 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:45:36.567768 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:45:36.567753 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:46:00.011288 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:00.011203 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7"] Jul 09 14:46:00.011824 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:00.011412 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" podUID="72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9" containerName="cleanup" containerID="cri-o://979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3" gracePeriod=30 Jul 09 14:46:30.185267 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.185246 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" Jul 09 14:46:30.301436 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.301367 2563 generic.go:356] "Generic (PLEG): container finished" podID="72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9" containerID="979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3" exitCode=137 Jul 09 14:46:30.301436 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.301411 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" Jul 09 14:46:30.301655 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.301448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" event={"ID":"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9","Type":"ContainerDied","Data":"979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3"} Jul 09 14:46:30.301655 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.301494 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7" event={"ID":"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9","Type":"ContainerDied","Data":"e08a3fcfba2de4c096f192ceecbf8ecc8c863c29e2e8522b92facd6ac2ce6255"} Jul 09 14:46:30.301655 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.301515 2563 scope.go:117] "RemoveContainer" containerID="979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3" Jul 09 14:46:30.309629 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.309601 2563 scope.go:117] "RemoveContainer" containerID="979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3" Jul 09 14:46:30.309884 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:46:30.309866 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3\": container with ID starting with 979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3 not found: ID does not exist" containerID="979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3" Jul 09 14:46:30.309935 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.309895 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3"} err="failed to get container status \"979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3\": rpc error: code = NotFound desc = could not find container \"979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3\": container with ID starting with 979b51656c895ad5a1f60cfa67d84b5096c32b97f175169f3afc78808609a8a3 not found: ID does not exist" Jul 09 14:46:30.321206 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.321183 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlhbb\" (UniqueName: \"kubernetes.io/projected/72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9-kube-api-access-rlhbb\") pod \"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9\" (UID: \"72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9\") " Jul 09 14:46:30.323057 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.323037 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9-kube-api-access-rlhbb" (OuterVolumeSpecName: "kube-api-access-rlhbb") pod "72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9" (UID: "72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9"). InnerVolumeSpecName "kube-api-access-rlhbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:46:30.422059 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.422035 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlhbb\" (UniqueName: \"kubernetes.io/projected/72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9-kube-api-access-rlhbb\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:46:30.619044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.619009 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7"] Jul 09 14:46:30.621188 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:30.621168 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726805-ccdz7"] Jul 09 14:46:32.573698 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:32.573671 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9" path="/var/lib/kubelet/pods/72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9/volumes" Jul 09 14:46:56.014334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.014302 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw"] Jul 09 14:46:56.014756 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.014692 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9" containerName="cleanup" Jul 09 14:46:56.014756 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.014708 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9" containerName="cleanup" Jul 09 14:46:56.014834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.014771 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72eaa91a-d8f2-4c5f-88b5-6237b1ebbdb9" containerName="cleanup" Jul 09 14:46:56.019020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.018994 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.022280 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.022260 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e66277d9-openshift-default-dockercfg-25jtp\"" Jul 09 14:46:56.035418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.035396 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw"] Jul 09 14:46:56.130039 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.129922 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk4ps\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-kube-api-access-qk4ps\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130055 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1231f2-7041-4a63-be28-3de896d849a3-istio-podinfo\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130157 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-envoy\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130178 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-istio-token\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1231f2-7041-4a63-be28-3de896d849a3-istiod-ca-cert\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130300 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-credential-socket\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130377 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-socket\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-certs\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.130437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.130430 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-data\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231337 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231304 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-socket\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231529 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-certs\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231529 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-data\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231529 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk4ps\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-kube-api-access-qk4ps\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231736 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1231f2-7041-4a63-be28-3de896d849a3-istio-podinfo\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231736 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-envoy\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231736 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231657 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-istio-token\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231736 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1231f2-7041-4a63-be28-3de896d849a3-istiod-ca-cert\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231951 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231769 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-certs\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.231951 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231777 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-credential-socket\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.232065 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.231986 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-data\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.232065 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.232029 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-socket\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.232172 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.232106 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-credential-socket\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.232497 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.232475 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1231f2-7041-4a63-be28-3de896d849a3-istiod-ca-cert\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.234353 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.234322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1231f2-7041-4a63-be28-3de896d849a3-istio-podinfo\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.234448 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.234430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-envoy\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.240130 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.240110 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-istio-token\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.240258 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.240147 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk4ps\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-kube-api-access-qk4ps\") pod \"e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.331457 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.331387 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:56.469714 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.469684 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw"] Jul 09 14:46:56.472990 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:46:56.472967 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1231f2_7041_4a63_be28_3de896d849a3.slice/crio-a66cc797eb34aa977b1c2a1f3d16d6767408dba149e1eb1f206d5bc7d1c11b1a WatchSource:0}: Error finding container a66cc797eb34aa977b1c2a1f3d16d6767408dba149e1eb1f206d5bc7d1c11b1a: Status 404 returned error can't find the container with id a66cc797eb34aa977b1c2a1f3d16d6767408dba149e1eb1f206d5bc7d1c11b1a Jul 09 14:46:56.474902 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.474873 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:46:56.474980 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.474937 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:46:56.474980 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:56.474965 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:46:57.427572 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.427534 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n"] Jul 09 14:46:57.431097 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.431074 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.434165 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.434138 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e66277d9-serving-cert\"" Jul 09 14:46:57.434512 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.434497 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:46:57.444068 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.444044 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" event={"ID":"df1231f2-7041-4a63-be28-3de896d849a3","Type":"ContainerStarted","Data":"577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889"} Jul 09 14:46:57.444161 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.444094 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" event={"ID":"df1231f2-7041-4a63-be28-3de896d849a3","Type":"ContainerStarted","Data":"a66cc797eb34aa977b1c2a1f3d16d6767408dba149e1eb1f206d5bc7d1c11b1a"} Jul 09 14:46:57.450571 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.450544 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n"] Jul 09 14:46:57.502580 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.502527 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" podStartSLOduration=2.502511643 podStartE2EDuration="2.502511643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:46:57.500395294 +0000 UTC m=+1581.521478559" watchObservedRunningTime="2026-07-09 14:46:57.502511643 +0000 UTC m=+1581.523594906" Jul 09 14:46:57.543948 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.543912 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjvw\" (UniqueName: \"kubernetes.io/projected/2736ceda-3a5a-439e-8c71-0e06b0045d20-kube-api-access-6sjvw\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.544138 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.544081 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2736ceda-3a5a-439e-8c71-0e06b0045d20-openshift-service-ca\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.544138 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.544124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2736ceda-3a5a-439e-8c71-0e06b0045d20-maas-api-tls\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.645258 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.645216 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2736ceda-3a5a-439e-8c71-0e06b0045d20-openshift-service-ca\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.645447 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.645264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2736ceda-3a5a-439e-8c71-0e06b0045d20-maas-api-tls\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.645447 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.645293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjvw\" (UniqueName: \"kubernetes.io/projected/2736ceda-3a5a-439e-8c71-0e06b0045d20-kube-api-access-6sjvw\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.646143 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.646113 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2736ceda-3a5a-439e-8c71-0e06b0045d20-openshift-service-ca\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.647810 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.647781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2736ceda-3a5a-439e-8c71-0e06b0045d20-maas-api-tls\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.659990 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.659963 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjvw\" (UniqueName: \"kubernetes.io/projected/2736ceda-3a5a-439e-8c71-0e06b0045d20-kube-api-access-6sjvw\") pod \"maas-api-e2e-ait-e66277d9-f8764f8d-lp69n\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.741848 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.741756 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:57.885922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:57.885895 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n"] Jul 09 14:46:57.887487 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:46:57.887463 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2736ceda_3a5a_439e_8c71_0e06b0045d20.slice/crio-a13abc3464c8ba90e7e76a3c670c88682536473af23dc90c4ddc6d73a43ad5ef WatchSource:0}: Error finding container a13abc3464c8ba90e7e76a3c670c88682536473af23dc90c4ddc6d73a43ad5ef: Status 404 returned error can't find the container with id a13abc3464c8ba90e7e76a3c670c88682536473af23dc90c4ddc6d73a43ad5ef Jul 09 14:46:58.332492 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:58.332457 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:58.337051 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:58.337027 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:58.450860 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:58.450824 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" event={"ID":"2736ceda-3a5a-439e-8c71-0e06b0045d20","Type":"ContainerStarted","Data":"a13abc3464c8ba90e7e76a3c670c88682536473af23dc90c4ddc6d73a43ad5ef"} Jul 09 14:46:58.451238 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:58.451090 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:58.452250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:58.452230 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:46:59.457483 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:59.457451 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" event={"ID":"2736ceda-3a5a-439e-8c71-0e06b0045d20","Type":"ContainerStarted","Data":"ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c"} Jul 09 14:46:59.457900 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:59.457524 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:46:59.477754 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:46:59.477704 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" podStartSLOduration=1.934347058 podStartE2EDuration="2.477688575s" totalImagesPullingTime="543.341517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:46:59.474446856 +0000 UTC m=+1583.495530119" watchObservedRunningTime="2026-07-09 14:46:59.477688575 +0000 UTC m=+1583.498771838" Jul 09 14:47:02.540688 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:02.540656 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw"] Jul 09 14:47:02.541129 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:02.540831 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" podUID="df1231f2-7041-4a63-be28-3de896d849a3" containerName="istio-proxy" containerID="cri-o://577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889" gracePeriod=30 Jul 09 14:47:02.919421 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:02.919345 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n"] Jul 09 14:47:02.919571 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:02.919512 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" podUID="2736ceda-3a5a-439e-8c71-0e06b0045d20" containerName="maas-api" containerID="cri-o://ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c" gracePeriod=30 Jul 09 14:47:03.214099 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.214074 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:47:03.396855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.396819 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sjvw\" (UniqueName: \"kubernetes.io/projected/2736ceda-3a5a-439e-8c71-0e06b0045d20-kube-api-access-6sjvw\") pod \"2736ceda-3a5a-439e-8c71-0e06b0045d20\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " Jul 09 14:47:03.397022 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.396914 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2736ceda-3a5a-439e-8c71-0e06b0045d20-openshift-service-ca\") pod \"2736ceda-3a5a-439e-8c71-0e06b0045d20\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " Jul 09 14:47:03.397022 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.396990 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2736ceda-3a5a-439e-8c71-0e06b0045d20-maas-api-tls\") pod \"2736ceda-3a5a-439e-8c71-0e06b0045d20\" (UID: \"2736ceda-3a5a-439e-8c71-0e06b0045d20\") " Jul 09 14:47:03.397312 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.397284 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2736ceda-3a5a-439e-8c71-0e06b0045d20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2736ceda-3a5a-439e-8c71-0e06b0045d20" (UID: "2736ceda-3a5a-439e-8c71-0e06b0045d20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:47:03.398952 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.398929 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2736ceda-3a5a-439e-8c71-0e06b0045d20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2736ceda-3a5a-439e-8c71-0e06b0045d20" (UID: "2736ceda-3a5a-439e-8c71-0e06b0045d20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:47:03.398952 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.398942 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2736ceda-3a5a-439e-8c71-0e06b0045d20-kube-api-access-6sjvw" (OuterVolumeSpecName: "kube-api-access-6sjvw") pod "2736ceda-3a5a-439e-8c71-0e06b0045d20" (UID: "2736ceda-3a5a-439e-8c71-0e06b0045d20"). InnerVolumeSpecName "kube-api-access-6sjvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:03.487652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.487524 2563 generic.go:356] "Generic (PLEG): container finished" podID="2736ceda-3a5a-439e-8c71-0e06b0045d20" containerID="ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c" exitCode=0 Jul 09 14:47:03.487652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.487562 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" event={"ID":"2736ceda-3a5a-439e-8c71-0e06b0045d20","Type":"ContainerDied","Data":"ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c"} Jul 09 14:47:03.487652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.487576 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" Jul 09 14:47:03.487941 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.487659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n" event={"ID":"2736ceda-3a5a-439e-8c71-0e06b0045d20","Type":"ContainerDied","Data":"a13abc3464c8ba90e7e76a3c670c88682536473af23dc90c4ddc6d73a43ad5ef"} Jul 09 14:47:03.487941 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.487682 2563 scope.go:117] "RemoveContainer" containerID="ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c" Jul 09 14:47:03.496677 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.496659 2563 scope.go:117] "RemoveContainer" containerID="ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c" Jul 09 14:47:03.496962 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:03.496939 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c\": container with ID starting with ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c not found: ID does not exist" containerID="ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c" Jul 09 14:47:03.497032 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.496975 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c"} err="failed to get container status \"ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c\": rpc error: code = NotFound desc = could not find container \"ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c\": container with ID starting with ada747b7432281b6765bcc08ae8076acc816818c157d15f9e1e082e52bb7542c not found: ID does not exist" Jul 09 14:47:03.498381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.498364 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2736ceda-3a5a-439e-8c71-0e06b0045d20-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:03.498444 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.498384 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2736ceda-3a5a-439e-8c71-0e06b0045d20-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:03.498444 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.498393 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjvw\" (UniqueName: \"kubernetes.io/projected/2736ceda-3a5a-439e-8c71-0e06b0045d20-kube-api-access-6sjvw\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:03.511199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.511170 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n"] Jul 09 14:47:03.519336 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:03.519311 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e66277d9-f8764f8d-lp69n"] Jul 09 14:47:04.574908 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:04.574878 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2736ceda-3a5a-439e-8c71-0e06b0045d20" path="/var/lib/kubelet/pods/2736ceda-3a5a-439e-8c71-0e06b0045d20/volumes" Jul 09 14:47:07.690201 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.690181 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:47:07.732254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732204 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-certs\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732238 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-istio-token\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732420 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732269 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1231f2-7041-4a63-be28-3de896d849a3-istio-podinfo\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732420 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732294 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-envoy\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732420 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732338 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-socket\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732420 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732366 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk4ps\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-kube-api-access-qk4ps\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732420 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732389 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1231f2-7041-4a63-be28-3de896d849a3-istiod-ca-cert\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732722 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732420 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-data\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732722 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732458 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-credential-socket\") pod \"df1231f2-7041-4a63-be28-3de896d849a3\" (UID: \"df1231f2-7041-4a63-be28-3de896d849a3\") " Jul 09 14:47:07.732722 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.732534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:07.733440 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.733280 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:07.735716 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.733573 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.735716 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.733602 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:07.735716 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.733921 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1231f2-7041-4a63-be28-3de896d849a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:47:07.735716 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.734265 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:07.735716 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.734921 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df1231f2-7041-4a63-be28-3de896d849a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:47:07.735716 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.734921 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:07.741645 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.741405 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:07.744113 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.744086 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-kube-api-access-qk4ps" (OuterVolumeSpecName: "kube-api-access-qk4ps") pod "df1231f2-7041-4a63-be28-3de896d849a3" (UID: "df1231f2-7041-4a63-be28-3de896d849a3"). InnerVolumeSpecName "kube-api-access-qk4ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:07.834658 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834636 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df1231f2-7041-4a63-be28-3de896d849a3-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.834658 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834659 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.834790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834668 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.834790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834677 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk4ps\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-kube-api-access-qk4ps\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.834790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834686 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df1231f2-7041-4a63-be28-3de896d849a3-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.834790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834694 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.834790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834701 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df1231f2-7041-4a63-be28-3de896d849a3-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:07.834790 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:07.834708 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df1231f2-7041-4a63-be28-3de896d849a3-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:08.517008 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.516980 2563 generic.go:356] "Generic (PLEG): container finished" podID="df1231f2-7041-4a63-be28-3de896d849a3" containerID="577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889" exitCode=0 Jul 09 14:47:08.517141 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.517034 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" Jul 09 14:47:08.517141 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.517072 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" event={"ID":"df1231f2-7041-4a63-be28-3de896d849a3","Type":"ContainerDied","Data":"577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889"} Jul 09 14:47:08.517223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.517159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw" event={"ID":"df1231f2-7041-4a63-be28-3de896d849a3","Type":"ContainerDied","Data":"a66cc797eb34aa977b1c2a1f3d16d6767408dba149e1eb1f206d5bc7d1c11b1a"} Jul 09 14:47:08.517223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.517188 2563 scope.go:117] "RemoveContainer" containerID="577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889" Jul 09 14:47:08.526053 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.526038 2563 scope.go:117] "RemoveContainer" containerID="577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889" Jul 09 14:47:08.526287 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:08.526271 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889\": container with ID starting with 577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889 not found: ID does not exist" containerID="577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889" Jul 09 14:47:08.526354 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.526292 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889"} err="failed to get container status \"577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889\": rpc error: code = NotFound desc = could not find container \"577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889\": container with ID starting with 577c90b232c641cd59286d2a54d172348e8ad7d3ac45ffc29b53d3770fec9889 not found: ID does not exist" Jul 09 14:47:08.540190 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.540169 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw"] Jul 09 14:47:08.546033 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.546010 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e66277d9-openshift-default-67d75cdddc-96hvw"] Jul 09 14:47:08.574401 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:08.574380 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df1231f2-7041-4a63-be28-3de896d849a3" path="/var/lib/kubelet/pods/df1231f2-7041-4a63-be28-3de896d849a3/volumes" Jul 09 14:47:09.422052 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.422022 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z"] Jul 09 14:47:09.422533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.422411 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df1231f2-7041-4a63-be28-3de896d849a3" containerName="istio-proxy" Jul 09 14:47:09.422533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.422425 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="df1231f2-7041-4a63-be28-3de896d849a3" containerName="istio-proxy" Jul 09 14:47:09.422533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.422456 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2736ceda-3a5a-439e-8c71-0e06b0045d20" containerName="maas-api" Jul 09 14:47:09.422533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.422462 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2736ceda-3a5a-439e-8c71-0e06b0045d20" containerName="maas-api" Jul 09 14:47:09.422533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.422515 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="df1231f2-7041-4a63-be28-3de896d849a3" containerName="istio-proxy" Jul 09 14:47:09.422533 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.422523 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2736ceda-3a5a-439e-8c71-0e06b0045d20" containerName="maas-api" Jul 09 14:47:09.425569 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.425549 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.428569 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.428550 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d6e2d797-openshift-default-dockercfg-86w89\"" Jul 09 14:47:09.441468 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.441449 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z"] Jul 09 14:47:09.447515 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447496 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istiod-ca-cert\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447597 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447532 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxhs\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-kube-api-access-fxxhs\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447597 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-credential-socket\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447692 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447603 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-socket\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447692 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447659 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-podinfo\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447692 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-envoy\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447803 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447695 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-data\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447803 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-certs\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.447803 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.447789 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-token\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.549720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.549692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-certs\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.549915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.549726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-token\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.549915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.549762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istiod-ca-cert\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.549915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.549804 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxhs\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-kube-api-access-fxxhs\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.549915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.549849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-credential-socket\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.549915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.549885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-socket\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550187 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.549922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-podinfo\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550187 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.550042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-envoy\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550187 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.550072 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-data\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550349 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.550244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-credential-socket\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550349 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.550296 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-socket\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550530 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.550509 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-data\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550648 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.550612 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-certs\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.550759 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.550738 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istiod-ca-cert\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.552797 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.552779 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-envoy\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.553010 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.552994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-podinfo\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.557895 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.557871 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-token\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.559150 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.559131 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxhs\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-kube-api-access-fxxhs\") pod \"e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.736272 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.736220 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:09.872714 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.872686 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z"] Jul 09 14:47:09.874084 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:47:09.874057 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1432b3a8_d3a1_4bb9_8778_85e8683110d4.slice/crio-8a54a991d5c10f4511fb9e9bd60aefa74db1c482b46ab8e5322e4a204f1e988a WatchSource:0}: Error finding container 8a54a991d5c10f4511fb9e9bd60aefa74db1c482b46ab8e5322e4a204f1e988a: Status 404 returned error can't find the container with id 8a54a991d5c10f4511fb9e9bd60aefa74db1c482b46ab8e5322e4a204f1e988a Jul 09 14:47:09.876246 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.876217 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:09.876305 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.876291 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:09.876346 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:09.876320 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:10.529698 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.529671 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" event={"ID":"1432b3a8-d3a1-4bb9-8778-85e8683110d4","Type":"ContainerStarted","Data":"4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61"} Jul 09 14:47:10.530052 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.529721 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" event={"ID":"1432b3a8-d3a1-4bb9-8778-85e8683110d4","Type":"ContainerStarted","Data":"8a54a991d5c10f4511fb9e9bd60aefa74db1c482b46ab8e5322e4a204f1e988a"} Jul 09 14:47:10.573381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.573337 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" podStartSLOduration=1.573320681 podStartE2EDuration="1.573320681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:47:10.570285897 +0000 UTC m=+1594.591369183" watchObservedRunningTime="2026-07-09 14:47:10.573320681 +0000 UTC m=+1594.594403944" Jul 09 14:47:10.658372 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.658335 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr"] Jul 09 14:47:10.660697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.660676 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.663471 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.663454 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:47:10.663644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.663609 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d6e2d797-serving-cert\"" Jul 09 14:47:10.676686 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.676665 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr"] Jul 09 14:47:10.737349 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.737325 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:10.741727 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.741709 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:10.760102 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.760081 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.760226 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.760118 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhnm\" (UniqueName: \"kubernetes.io/projected/b7285661-9c5c-4549-a551-a75ba323f5b8-kube-api-access-9qhnm\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.760226 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.760153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7285661-9c5c-4549-a551-a75ba323f5b8-openshift-service-ca\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.861075 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.860997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhnm\" (UniqueName: \"kubernetes.io/projected/b7285661-9c5c-4549-a551-a75ba323f5b8-kube-api-access-9qhnm\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.861075 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.861039 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7285661-9c5c-4549-a551-a75ba323f5b8-openshift-service-ca\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.861290 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.861094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.861290 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:10.861172 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-serving-cert: secret "maas-api-e2e-ait-d6e2d797-serving-cert" not found Jul 09 14:47:10.861290 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:10.861221 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls podName:b7285661-9c5c-4549-a551-a75ba323f5b8 nodeName:}" failed. No retries permitted until 2026-07-09 14:47:11.361205406 +0000 UTC m=+1595.382288649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls") pod "maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" (UID: "b7285661-9c5c-4549-a551-a75ba323f5b8") : secret "maas-api-e2e-ait-d6e2d797-serving-cert" not found Jul 09 14:47:10.861737 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.861720 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7285661-9c5c-4549-a551-a75ba323f5b8-openshift-service-ca\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:10.872369 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:10.872349 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhnm\" (UniqueName: \"kubernetes.io/projected/b7285661-9c5c-4549-a551-a75ba323f5b8-kube-api-access-9qhnm\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:11.366336 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:11.366298 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:11.368711 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:11.368684 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls\") pod \"maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:11.534770 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:11.534737 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:11.535798 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:11.535776 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:11.571349 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:11.571299 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:11.712700 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:11.712648 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr"] Jul 09 14:47:11.714274 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:47:11.714250 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7285661_9c5c_4549_a551_a75ba323f5b8.slice/crio-0e0f534eaaf18df44abcab7e8c90431751bac354e80ec78d5284efc92ab2045d WatchSource:0}: Error finding container 0e0f534eaaf18df44abcab7e8c90431751bac354e80ec78d5284efc92ab2045d: Status 404 returned error can't find the container with id 0e0f534eaaf18df44abcab7e8c90431751bac354e80ec78d5284efc92ab2045d Jul 09 14:47:12.541438 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:12.541405 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" event={"ID":"b7285661-9c5c-4549-a551-a75ba323f5b8","Type":"ContainerStarted","Data":"a20d792d4fbb3af3f3b921a69c829ab61139602dca45c5515f77289203361f08"} Jul 09 14:47:12.541800 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:12.541461 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" event={"ID":"b7285661-9c5c-4549-a551-a75ba323f5b8","Type":"ContainerStarted","Data":"0e0f534eaaf18df44abcab7e8c90431751bac354e80ec78d5284efc92ab2045d"} Jul 09 14:47:12.541800 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:12.541482 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:12.561880 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:12.561827 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" podStartSLOduration=2.192599654 podStartE2EDuration="2.561811348s" totalImagesPullingTime="369.211694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:47:12.560843649 +0000 UTC m=+1596.581926914" watchObservedRunningTime="2026-07-09 14:47:12.561811348 +0000 UTC m=+1596.582894611" Jul 09 14:47:16.326151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.326120 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr"] Jul 09 14:47:16.326563 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.326295 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" podUID="b7285661-9c5c-4549-a551-a75ba323f5b8" containerName="maas-api" containerID="cri-o://a20d792d4fbb3af3f3b921a69c829ab61139602dca45c5515f77289203361f08" gracePeriod=30 Jul 09 14:47:16.585245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.585163 2563 generic.go:356] "Generic (PLEG): container finished" podID="b7285661-9c5c-4549-a551-a75ba323f5b8" containerID="a20d792d4fbb3af3f3b921a69c829ab61139602dca45c5515f77289203361f08" exitCode=0 Jul 09 14:47:16.585453 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.585435 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" event={"ID":"b7285661-9c5c-4549-a551-a75ba323f5b8","Type":"ContainerDied","Data":"a20d792d4fbb3af3f3b921a69c829ab61139602dca45c5515f77289203361f08"} Jul 09 14:47:16.602715 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.602689 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:16.714819 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.714787 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7285661-9c5c-4549-a551-a75ba323f5b8-openshift-service-ca\") pod \"b7285661-9c5c-4549-a551-a75ba323f5b8\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " Jul 09 14:47:16.714992 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.714856 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls\") pod \"b7285661-9c5c-4549-a551-a75ba323f5b8\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " Jul 09 14:47:16.714992 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.714889 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qhnm\" (UniqueName: \"kubernetes.io/projected/b7285661-9c5c-4549-a551-a75ba323f5b8-kube-api-access-9qhnm\") pod \"b7285661-9c5c-4549-a551-a75ba323f5b8\" (UID: \"b7285661-9c5c-4549-a551-a75ba323f5b8\") " Jul 09 14:47:16.715159 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.715138 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7285661-9c5c-4549-a551-a75ba323f5b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7285661-9c5c-4549-a551-a75ba323f5b8" (UID: "b7285661-9c5c-4549-a551-a75ba323f5b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:47:16.716995 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.716969 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7285661-9c5c-4549-a551-a75ba323f5b8-kube-api-access-9qhnm" (OuterVolumeSpecName: "kube-api-access-9qhnm") pod "b7285661-9c5c-4549-a551-a75ba323f5b8" (UID: "b7285661-9c5c-4549-a551-a75ba323f5b8"). InnerVolumeSpecName "kube-api-access-9qhnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:16.717092 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.716995 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7285661-9c5c-4549-a551-a75ba323f5b8" (UID: "b7285661-9c5c-4549-a551-a75ba323f5b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:47:16.816440 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.816412 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7285661-9c5c-4549-a551-a75ba323f5b8-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:16.816440 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.816435 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7285661-9c5c-4549-a551-a75ba323f5b8-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:16.816440 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:16.816444 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhnm\" (UniqueName: \"kubernetes.io/projected/b7285661-9c5c-4549-a551-a75ba323f5b8-kube-api-access-9qhnm\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:17.592209 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:17.592179 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" Jul 09 14:47:17.592209 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:17.592190 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr" event={"ID":"b7285661-9c5c-4549-a551-a75ba323f5b8","Type":"ContainerDied","Data":"0e0f534eaaf18df44abcab7e8c90431751bac354e80ec78d5284efc92ab2045d"} Jul 09 14:47:17.592737 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:17.592242 2563 scope.go:117] "RemoveContainer" containerID="a20d792d4fbb3af3f3b921a69c829ab61139602dca45c5515f77289203361f08" Jul 09 14:47:17.614590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:17.614556 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr"] Jul 09 14:47:17.619106 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:17.619079 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d6e2d797-5f558c5746-gcxvr"] Jul 09 14:47:18.575298 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:18.575265 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7285661-9c5c-4549-a551-a75ba323f5b8" path="/var/lib/kubelet/pods/b7285661-9c5c-4549-a551-a75ba323f5b8/volumes" Jul 09 14:47:21.616437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:21.616350 2563 generic.go:356] "Generic (PLEG): container finished" podID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerID="4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61" exitCode=0 Jul 09 14:47:21.616437 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:21.616413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" event={"ID":"1432b3a8-d3a1-4bb9-8778-85e8683110d4","Type":"ContainerDied","Data":"4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61"} Jul 09 14:47:21.617262 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:21.616906 2563 scope.go:117] "RemoveContainer" containerID="4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61" Jul 09 14:47:21.617924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:21.617897 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:21.618033 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:21.617957 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:21.618033 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:21.617992 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:21.930944 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:21.930866 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z"] Jul 09 14:47:22.623121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:22.623073 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" event={"ID":"1432b3a8-d3a1-4bb9-8778-85e8683110d4","Type":"ContainerStarted","Data":"a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6"} Jul 09 14:47:22.623572 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:22.623169 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerName="istio-proxy" containerID="cri-o://a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6" gracePeriod=30 Jul 09 14:47:26.534706 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:26.534667 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:27.878784 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:27.878763 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:28.015261 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015239 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-certs\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015384 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015286 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-data\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015384 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015309 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-token\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015384 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015337 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-envoy\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015384 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015357 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istiod-ca-cert\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015384 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015376 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxxhs\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-kube-api-access-fxxhs\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015395 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-socket\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015427 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-credential-socket\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015471 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-podinfo\") pod \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\" (UID: \"1432b3a8-d3a1-4bb9-8778-85e8683110d4\") " Jul 09 14:47:28.015608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015531 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:28.015840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015763 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.015906 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015839 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:28.015977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.015950 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:47:28.016092 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.016071 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:28.016155 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.016143 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:28.017640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.017589 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:47:28.017640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.017593 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:28.017793 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.017763 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:28.018081 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.018061 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-kube-api-access-fxxhs" (OuterVolumeSpecName: "kube-api-access-fxxhs") pod "1432b3a8-d3a1-4bb9-8778-85e8683110d4" (UID: "1432b3a8-d3a1-4bb9-8778-85e8683110d4"). InnerVolumeSpecName "kube-api-access-fxxhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:28.116492 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116472 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.116605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116494 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.116605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116508 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.116605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116520 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.116605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116530 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.116605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116541 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1432b3a8-d3a1-4bb9-8778-85e8683110d4-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.116605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116552 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxhs\" (UniqueName: \"kubernetes.io/projected/1432b3a8-d3a1-4bb9-8778-85e8683110d4-kube-api-access-fxxhs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.116605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.116565 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1432b3a8-d3a1-4bb9-8778-85e8683110d4-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:28.159593 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.159533 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2"] Jul 09 14:47:28.160085 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.160065 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7285661-9c5c-4549-a551-a75ba323f5b8" containerName="maas-api" Jul 09 14:47:28.160229 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.160088 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7285661-9c5c-4549-a551-a75ba323f5b8" containerName="maas-api" Jul 09 14:47:28.160229 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.160121 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerName="istio-proxy" Jul 09 14:47:28.160229 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.160130 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerName="istio-proxy" Jul 09 14:47:28.160229 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.160227 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerName="istio-proxy" Jul 09 14:47:28.160434 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.160242 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7285661-9c5c-4549-a551-a75ba323f5b8" containerName="maas-api" Jul 09 14:47:28.162802 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.162783 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.165992 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.165951 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f1a0c39c-openshift-default-dockercfg-m8rrn\"" Jul 09 14:47:28.181028 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.181008 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2"] Jul 09 14:47:28.217456 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217428 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-envoy\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217556 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-podinfo\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217556 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217499 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-token\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217549 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istiod-ca-cert\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-socket\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217652 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-data\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217765 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217666 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47czj\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-kube-api-access-47czj\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217765 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-certs\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.217838 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.217767 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-credential-socket\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318463 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318408 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-envoy\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318463 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-podinfo\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318463 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318460 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-token\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istiod-ca-cert\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318510 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-socket\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318536 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-data\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318556 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47czj\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-kube-api-access-47czj\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318601 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-certs\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.318708 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-credential-socket\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.319012 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.318991 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-data\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.319074 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.319006 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-credential-socket\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.319129 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.319095 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-certs\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.319203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.319182 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-socket\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.319306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.319290 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istiod-ca-cert\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.320694 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.320678 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-envoy\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.320806 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.320791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-podinfo\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.327939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.327920 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-token\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.328243 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.328224 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47czj\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-kube-api-access-47czj\") pod \"e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.475378 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.475356 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:28.606465 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.606440 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2"] Jul 09 14:47:28.607948 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:47:28.607917 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafb0c85_dbba_48cf_a50a_a88f3fe81d9f.slice/crio-3f78687dc9c01b149c1f9edbe1f6ca0eb3050d16d4a2c5e6031b5f9915fd5d70 WatchSource:0}: Error finding container 3f78687dc9c01b149c1f9edbe1f6ca0eb3050d16d4a2c5e6031b5f9915fd5d70: Status 404 returned error can't find the container with id 3f78687dc9c01b149c1f9edbe1f6ca0eb3050d16d4a2c5e6031b5f9915fd5d70 Jul 09 14:47:28.609841 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.609813 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:28.609915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.609878 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:28.609915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.609904 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:47:28.660590 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.660568 2563 generic.go:356] "Generic (PLEG): container finished" podID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerID="a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6" exitCode=0 Jul 09 14:47:28.660695 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.660609 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" Jul 09 14:47:28.660695 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.660650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" event={"ID":"1432b3a8-d3a1-4bb9-8778-85e8683110d4","Type":"ContainerDied","Data":"a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6"} Jul 09 14:47:28.660816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.660702 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z" event={"ID":"1432b3a8-d3a1-4bb9-8778-85e8683110d4","Type":"ContainerDied","Data":"8a54a991d5c10f4511fb9e9bd60aefa74db1c482b46ab8e5322e4a204f1e988a"} Jul 09 14:47:28.660816 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.660728 2563 scope.go:117] "RemoveContainer" containerID="a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6" Jul 09 14:47:28.661813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.661796 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" event={"ID":"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f","Type":"ContainerStarted","Data":"3f78687dc9c01b149c1f9edbe1f6ca0eb3050d16d4a2c5e6031b5f9915fd5d70"} Jul 09 14:47:28.670730 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.670710 2563 scope.go:117] "RemoveContainer" containerID="4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61" Jul 09 14:47:28.678378 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.678355 2563 scope.go:117] "RemoveContainer" containerID="a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6" Jul 09 14:47:28.679186 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:28.679158 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6\": container with ID starting with a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6 not found: ID does not exist" containerID="a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6" Jul 09 14:47:28.679318 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.679196 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6"} err="failed to get container status \"a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6\": rpc error: code = NotFound desc = could not find container \"a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6\": container with ID starting with a3e9920b57cc49c33ac9c8902733016579ada28cfd862464add33ea3896a2dc6 not found: ID does not exist" Jul 09 14:47:28.679318 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.679218 2563 scope.go:117] "RemoveContainer" containerID="4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61" Jul 09 14:47:28.679515 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:28.679491 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61\": container with ID starting with 4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61 not found: ID does not exist" containerID="4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61" Jul 09 14:47:28.679830 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.679520 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61"} err="failed to get container status \"4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61\": rpc error: code = NotFound desc = could not find container \"4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61\": container with ID starting with 4eb2169144825bd5c37f2ae085079e270abfbcf02d39cea79ca28931144a6e61 not found: ID does not exist" Jul 09 14:47:28.681717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.681698 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z"] Jul 09 14:47:28.686174 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:28.686156 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d6e2d797-openshift-default-7c6ff8b4ff-tm76z"] Jul 09 14:47:29.460416 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.460382 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc"] Jul 09 14:47:29.460900 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.460880 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerName="istio-proxy" Jul 09 14:47:29.460976 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.460905 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerName="istio-proxy" Jul 09 14:47:29.461042 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.461028 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" containerName="istio-proxy" Jul 09 14:47:29.463146 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.463126 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.465911 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.465887 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f1a0c39c-serving-cert\"" Jul 09 14:47:29.466012 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.465894 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:47:29.475044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.475018 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc"] Jul 09 14:47:29.528775 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.528750 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2w87\" (UniqueName: \"kubernetes.io/projected/d0edbfb1-452d-4eb9-b976-6890c4920bb3-kube-api-access-l2w87\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.528879 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.528791 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.528942 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.528878 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0edbfb1-452d-4eb9-b976-6890c4920bb3-openshift-service-ca\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.630055 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.630021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.630233 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.630081 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0edbfb1-452d-4eb9-b976-6890c4920bb3-openshift-service-ca\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.630233 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.630112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2w87\" (UniqueName: \"kubernetes.io/projected/d0edbfb1-452d-4eb9-b976-6890c4920bb3-kube-api-access-l2w87\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.630233 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:29.630203 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-serving-cert: secret "maas-api-e2e-derive-f1a0c39c-serving-cert" not found Jul 09 14:47:29.630369 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:29.630282 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls podName:d0edbfb1-452d-4eb9-b976-6890c4920bb3 nodeName:}" failed. No retries permitted until 2026-07-09 14:47:30.130257798 +0000 UTC m=+1614.151341055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls") pod "maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" (UID: "d0edbfb1-452d-4eb9-b976-6890c4920bb3") : secret "maas-api-e2e-derive-f1a0c39c-serving-cert" not found Jul 09 14:47:29.630776 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.630756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0edbfb1-452d-4eb9-b976-6890c4920bb3-openshift-service-ca\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.639926 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.639900 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2w87\" (UniqueName: \"kubernetes.io/projected/d0edbfb1-452d-4eb9-b976-6890c4920bb3-kube-api-access-l2w87\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:29.668808 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.668775 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" event={"ID":"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f","Type":"ContainerStarted","Data":"2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a"} Jul 09 14:47:29.690889 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:29.690844 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" podStartSLOduration=1.690830858 podStartE2EDuration="1.690830858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:47:29.688805726 +0000 UTC m=+1613.709889012" watchObservedRunningTime="2026-07-09 14:47:29.690830858 +0000 UTC m=+1613.711914172" Jul 09 14:47:30.134327 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.134299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:30.136721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.136693 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls\") pod \"maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:30.373359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.373326 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:30.482807 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.482758 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:30.488569 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.488491 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:30.531505 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.531483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc"] Jul 09 14:47:30.533109 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:47:30.533086 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0edbfb1_452d_4eb9_b976_6890c4920bb3.slice/crio-c870208fc1fdf41076a519edcece02eaf2f9a3b21de4f3e180b8fceaefe30e5f WatchSource:0}: Error finding container c870208fc1fdf41076a519edcece02eaf2f9a3b21de4f3e180b8fceaefe30e5f: Status 404 returned error can't find the container with id c870208fc1fdf41076a519edcece02eaf2f9a3b21de4f3e180b8fceaefe30e5f Jul 09 14:47:30.574910 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.574884 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1432b3a8-d3a1-4bb9-8778-85e8683110d4" path="/var/lib/kubelet/pods/1432b3a8-d3a1-4bb9-8778-85e8683110d4/volumes" Jul 09 14:47:30.676411 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.676333 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" event={"ID":"d0edbfb1-452d-4eb9-b976-6890c4920bb3","Type":"ContainerStarted","Data":"c870208fc1fdf41076a519edcece02eaf2f9a3b21de4f3e180b8fceaefe30e5f"} Jul 09 14:47:30.676613 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.676603 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:30.677382 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:30.677366 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:31.683994 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:31.683953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" event={"ID":"d0edbfb1-452d-4eb9-b976-6890c4920bb3","Type":"ContainerStarted","Data":"2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65"} Jul 09 14:47:31.703991 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:31.703934 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" podStartSLOduration=2.228004669 podStartE2EDuration="2.703914078s" totalImagesPullingTime="475.909409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:47:31.703280587 +0000 UTC m=+1615.724363845" watchObservedRunningTime="2026-07-09 14:47:31.703914078 +0000 UTC m=+1615.724997341" Jul 09 14:47:32.689271 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:32.689235 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:33.927743 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:33.927708 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2"] Jul 09 14:47:33.928120 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:33.927887 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" podUID="eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" containerName="istio-proxy" containerID="cri-o://2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a" gracePeriod=30 Jul 09 14:47:34.191980 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:34.191890 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc"] Jul 09 14:47:34.699746 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:34.699709 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" podUID="d0edbfb1-452d-4eb9-b976-6890c4920bb3" containerName="maas-api" containerID="cri-o://2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65" gracePeriod=30 Jul 09 14:47:34.979333 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:34.979303 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:35.081822 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.081790 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls\") pod \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " Jul 09 14:47:35.081990 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.081842 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0edbfb1-452d-4eb9-b976-6890c4920bb3-openshift-service-ca\") pod \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " Jul 09 14:47:35.081990 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.081878 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2w87\" (UniqueName: \"kubernetes.io/projected/d0edbfb1-452d-4eb9-b976-6890c4920bb3-kube-api-access-l2w87\") pod \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\" (UID: \"d0edbfb1-452d-4eb9-b976-6890c4920bb3\") " Jul 09 14:47:35.082175 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.082151 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0edbfb1-452d-4eb9-b976-6890c4920bb3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0edbfb1-452d-4eb9-b976-6890c4920bb3" (UID: "d0edbfb1-452d-4eb9-b976-6890c4920bb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:47:35.083878 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.083852 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0edbfb1-452d-4eb9-b976-6890c4920bb3" (UID: "d0edbfb1-452d-4eb9-b976-6890c4920bb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:47:35.083982 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.083933 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0edbfb1-452d-4eb9-b976-6890c4920bb3-kube-api-access-l2w87" (OuterVolumeSpecName: "kube-api-access-l2w87") pod "d0edbfb1-452d-4eb9-b976-6890c4920bb3" (UID: "d0edbfb1-452d-4eb9-b976-6890c4920bb3"). InnerVolumeSpecName "kube-api-access-l2w87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:35.182530 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.182497 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0edbfb1-452d-4eb9-b976-6890c4920bb3-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:35.182530 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.182531 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0edbfb1-452d-4eb9-b976-6890c4920bb3-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:35.182530 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.182546 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2w87\" (UniqueName: \"kubernetes.io/projected/d0edbfb1-452d-4eb9-b976-6890c4920bb3-kube-api-access-l2w87\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:35.706303 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.706265 2563 generic.go:356] "Generic (PLEG): container finished" podID="d0edbfb1-452d-4eb9-b976-6890c4920bb3" containerID="2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65" exitCode=0 Jul 09 14:47:35.706473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.706335 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" Jul 09 14:47:35.706473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.706354 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" event={"ID":"d0edbfb1-452d-4eb9-b976-6890c4920bb3","Type":"ContainerDied","Data":"2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65"} Jul 09 14:47:35.706473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.706405 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc" event={"ID":"d0edbfb1-452d-4eb9-b976-6890c4920bb3","Type":"ContainerDied","Data":"c870208fc1fdf41076a519edcece02eaf2f9a3b21de4f3e180b8fceaefe30e5f"} Jul 09 14:47:35.706473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.706428 2563 scope.go:117] "RemoveContainer" containerID="2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65" Jul 09 14:47:35.715744 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.715726 2563 scope.go:117] "RemoveContainer" containerID="2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65" Jul 09 14:47:35.715968 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:35.715949 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65\": container with ID starting with 2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65 not found: ID does not exist" containerID="2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65" Jul 09 14:47:35.716025 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.715974 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65"} err="failed to get container status \"2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65\": rpc error: code = NotFound desc = could not find container \"2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65\": container with ID starting with 2f6c5a31af859229ce103f4b67ca4c681db1cf53cc43196e795ec82412701f65 not found: ID does not exist" Jul 09 14:47:35.730734 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.730709 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc"] Jul 09 14:47:35.734603 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:35.734584 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f1a0c39c-6f97f56f5f-6p9qc"] Jul 09 14:47:36.576579 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:36.576540 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0edbfb1-452d-4eb9-b976-6890c4920bb3" path="/var/lib/kubelet/pods/d0edbfb1-452d-4eb9-b976-6890c4920bb3/volumes" Jul 09 14:47:39.874326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:39.874302 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:40.024105 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024076 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-data\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024153 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-envoy\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024181 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47czj\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-kube-api-access-47czj\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024201 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-podinfo\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024245 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024237 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istiod-ca-cert\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024461 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024315 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-credential-socket\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024461 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024369 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-certs\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024461 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024415 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-token\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024461 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024447 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-socket\") pod \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\" (UID: \"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f\") " Jul 09 14:47:40.024687 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024558 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:40.024787 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024765 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:40.024855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024784 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.024855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024783 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:40.024978 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.024950 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:47:40.025049 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.025025 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:40.026429 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.026405 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-kube-api-access-47czj" (OuterVolumeSpecName: "kube-api-access-47czj") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "kube-api-access-47czj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:40.026543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.026454 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:47:40.026543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.026460 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:47:40.026761 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.026746 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" (UID: "eafb0c85-dbba-48cf-a50a-a88f3fe81d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:47:40.125247 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125226 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.125247 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125245 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.125385 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125254 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.125385 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125263 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47czj\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-kube-api-access-47czj\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.125385 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125271 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.125385 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125279 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.125385 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125286 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.125385 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.125293 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:47:40.733813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.733782 2563 generic.go:356] "Generic (PLEG): container finished" podID="eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" containerID="2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a" exitCode=0 Jul 09 14:47:40.733975 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.733835 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" Jul 09 14:47:40.733975 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.733837 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" event={"ID":"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f","Type":"ContainerDied","Data":"2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a"} Jul 09 14:47:40.734046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.733986 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2" event={"ID":"eafb0c85-dbba-48cf-a50a-a88f3fe81d9f","Type":"ContainerDied","Data":"3f78687dc9c01b149c1f9edbe1f6ca0eb3050d16d4a2c5e6031b5f9915fd5d70"} Jul 09 14:47:40.734046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.734016 2563 scope.go:117] "RemoveContainer" containerID="2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a" Jul 09 14:47:40.743102 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.742930 2563 scope.go:117] "RemoveContainer" containerID="2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a" Jul 09 14:47:40.743222 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:47:40.743205 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a\": container with ID starting with 2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a not found: ID does not exist" containerID="2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a" Jul 09 14:47:40.743270 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.743231 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a"} err="failed to get container status \"2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a\": rpc error: code = NotFound desc = could not find container \"2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a\": container with ID starting with 2389c364dffaa5e9d61e8acd40429741de8852176df7e80699fe64dc9e913b7a not found: ID does not exist" Jul 09 14:47:40.779879 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.779847 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2"] Jul 09 14:47:40.798343 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:40.798310 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f1a0c39c-openshift-default-5b658f65c9-lddf2"] Jul 09 14:47:42.574449 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:47:42.574413 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" path="/var/lib/kubelet/pods/eafb0c85-dbba-48cf-a50a-a88f3fe81d9f/volumes" Jul 09 14:49:20.576780 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.576693 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j"] Jul 09 14:49:20.577287 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.577184 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" containerName="istio-proxy" Jul 09 14:49:20.577287 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.577199 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" containerName="istio-proxy" Jul 09 14:49:20.577287 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.577224 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0edbfb1-452d-4eb9-b976-6890c4920bb3" containerName="maas-api" Jul 09 14:49:20.577287 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.577232 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0edbfb1-452d-4eb9-b976-6890c4920bb3" containerName="maas-api" Jul 09 14:49:20.577422 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.577318 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafb0c85-dbba-48cf-a50a-a88f3fe81d9f" containerName="istio-proxy" Jul 09 14:49:20.577422 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.577334 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0edbfb1-452d-4eb9-b976-6890c4920bb3" containerName="maas-api" Jul 09 14:49:20.580709 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.580690 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.591035 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.591017 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3abd8947-openshift-default-dockercfg-gs78m\"" Jul 09 14:49:20.698581 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698551 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-podinfo\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698719 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698593 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-data\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698719 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698689 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-certs\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698807 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764e6fc-0fff-4b29-bfb8-7161354233aa-istiod-ca-cert\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698807 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698778 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-socket\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698807 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-token\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698984 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-envoy\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698984 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2rf\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-kube-api-access-hw2rf\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.698984 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.698923 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-credential-socket\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.724382 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.724358 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j"] Jul 09 14:49:20.799335 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799308 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764e6fc-0fff-4b29-bfb8-7161354233aa-istiod-ca-cert\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799447 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799341 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-socket\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799447 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-token\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799447 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799422 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-envoy\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799609 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2rf\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-kube-api-access-hw2rf\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799685 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799639 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-credential-socket\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799685 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799681 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-podinfo\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799785 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-data\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799785 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-certs\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.799893 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.799787 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-socket\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.800028 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.800002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-credential-socket\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.800155 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.800049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-data\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.800155 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.800059 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764e6fc-0fff-4b29-bfb8-7161354233aa-istiod-ca-cert\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.800155 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.800129 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-certs\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.801974 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.801949 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-envoy\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.802322 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.802253 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-podinfo\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.808749 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.808726 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-token\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.809700 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.809676 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2rf\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-kube-api-access-hw2rf\") pod \"e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:20.892492 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:20.892443 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:21.052441 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.052418 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j"] Jul 09 14:49:21.055109 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:49:21.055076 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2764e6fc_0fff_4b29_bfb8_7161354233aa.slice/crio-d1da0318cc7eb101c9e5c85aca365d78bccaa2687ccd47e755e2f679dc595d3f WatchSource:0}: Error finding container d1da0318cc7eb101c9e5c85aca365d78bccaa2687ccd47e755e2f679dc595d3f: Status 404 returned error can't find the container with id d1da0318cc7eb101c9e5c85aca365d78bccaa2687ccd47e755e2f679dc595d3f Jul 09 14:49:21.057634 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.057583 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:49:21.057705 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.057689 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:49:21.057746 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.057734 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:49:21.247875 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.247784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" event={"ID":"2764e6fc-0fff-4b29-bfb8-7161354233aa","Type":"ContainerStarted","Data":"c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755"} Jul 09 14:49:21.247875 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.247851 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" event={"ID":"2764e6fc-0fff-4b29-bfb8-7161354233aa","Type":"ContainerStarted","Data":"d1da0318cc7eb101c9e5c85aca365d78bccaa2687ccd47e755e2f679dc595d3f"} Jul 09 14:49:21.287136 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.287080 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" podStartSLOduration=1.287060976 podStartE2EDuration="1.287060976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:21.283870641 +0000 UTC m=+1725.304953925" watchObservedRunningTime="2026-07-09 14:49:21.287060976 +0000 UTC m=+1725.308144239" Jul 09 14:49:21.892833 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:21.892807 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:22.898840 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:22.898799 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:23.259847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:23.259813 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:23.260801 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:23.260778 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:24.108898 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.108870 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58"] Jul 09 14:49:24.112342 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.112323 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.116783 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.116760 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3abd8947-serving-cert\"" Jul 09 14:49:24.117059 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.117043 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:49:24.150749 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.150723 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58"] Jul 09 14:49:24.233915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.233885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hhq\" (UniqueName: \"kubernetes.io/projected/0ebe245d-800f-49e8-aa24-05960b155f65-kube-api-access-v9hhq\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.234132 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.233940 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe245d-800f-49e8-aa24-05960b155f65-openshift-service-ca\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.234132 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.234033 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.334803 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.334779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.334905 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.334859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hhq\" (UniqueName: \"kubernetes.io/projected/0ebe245d-800f-49e8-aa24-05960b155f65-kube-api-access-v9hhq\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.334966 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.334903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe245d-800f-49e8-aa24-05960b155f65-openshift-service-ca\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.334966 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:49:24.334911 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-serving-cert: secret "maas-api-e2e-mt-3abd8947-serving-cert" not found Jul 09 14:49:24.335081 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:49:24.335002 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls podName:0ebe245d-800f-49e8-aa24-05960b155f65 nodeName:}" failed. No retries permitted until 2026-07-09 14:49:24.834982516 +0000 UTC m=+1728.856065768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls") pod "maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" (UID: "0ebe245d-800f-49e8-aa24-05960b155f65") : secret "maas-api-e2e-mt-3abd8947-serving-cert" not found Jul 09 14:49:24.335508 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.335489 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe245d-800f-49e8-aa24-05960b155f65-openshift-service-ca\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.379741 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.379681 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hhq\" (UniqueName: \"kubernetes.io/projected/0ebe245d-800f-49e8-aa24-05960b155f65-kube-api-access-v9hhq\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.838565 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.838532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:24.841158 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:24.841130 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls\") pod \"maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:25.022691 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:25.022661 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:25.188847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:25.188824 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58"] Jul 09 14:49:25.190488 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:49:25.190465 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebe245d_800f_49e8_aa24_05960b155f65.slice/crio-ab7e1d7c71ceb6ad2cd7a1b286ce385510a051d639d89c8a4263618b4388ecee WatchSource:0}: Error finding container ab7e1d7c71ceb6ad2cd7a1b286ce385510a051d639d89c8a4263618b4388ecee: Status 404 returned error can't find the container with id ab7e1d7c71ceb6ad2cd7a1b286ce385510a051d639d89c8a4263618b4388ecee Jul 09 14:49:25.271766 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:25.271734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" event={"ID":"0ebe245d-800f-49e8-aa24-05960b155f65","Type":"ContainerStarted","Data":"ab7e1d7c71ceb6ad2cd7a1b286ce385510a051d639d89c8a4263618b4388ecee"} Jul 09 14:49:26.279258 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:26.279219 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" event={"ID":"0ebe245d-800f-49e8-aa24-05960b155f65","Type":"ContainerStarted","Data":"0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7"} Jul 09 14:49:26.279606 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:26.279307 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:26.337372 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:26.337325 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" podStartSLOduration=1.612139818 podStartE2EDuration="2.337309458s" totalImagesPullingTime="725.16964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:49:26.335453307 +0000 UTC m=+1730.356536572" watchObservedRunningTime="2026-07-09 14:49:26.337309458 +0000 UTC m=+1730.358392721" Jul 09 14:49:32.289356 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:32.289324 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:37.344555 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:37.344519 2563 generic.go:356] "Generic (PLEG): container finished" podID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerID="c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755" exitCode=0 Jul 09 14:49:37.344993 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:37.344592 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" event={"ID":"2764e6fc-0fff-4b29-bfb8-7161354233aa","Type":"ContainerDied","Data":"c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755"} Jul 09 14:49:37.345044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:37.345023 2563 scope.go:117] "RemoveContainer" containerID="c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755" Jul 09 14:49:37.346051 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:37.346022 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:49:37.346110 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:37.346092 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:49:37.346152 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:37.346122 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:49:38.260375 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.260346 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:38.352237 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.352208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" event={"ID":"2764e6fc-0fff-4b29-bfb8-7161354233aa","Type":"ContainerStarted","Data":"629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd"} Jul 09 14:49:38.582345 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.582274 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58"] Jul 09 14:49:38.582482 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.582449 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" podUID="0ebe245d-800f-49e8-aa24-05960b155f65" containerName="maas-api" containerID="cri-o://0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7" gracePeriod=30 Jul 09 14:49:38.843613 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.843558 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:38.893207 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.893162 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:38.894504 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.894476 2563 patch_prober.go:28] interesting pod/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 14:49:38.894639 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.894529 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 14:49:38.970387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.970356 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe245d-800f-49e8-aa24-05960b155f65-openshift-service-ca\") pod \"0ebe245d-800f-49e8-aa24-05960b155f65\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " Jul 09 14:49:38.970586 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.970400 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls\") pod \"0ebe245d-800f-49e8-aa24-05960b155f65\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " Jul 09 14:49:38.970586 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.970488 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9hhq\" (UniqueName: \"kubernetes.io/projected/0ebe245d-800f-49e8-aa24-05960b155f65-kube-api-access-v9hhq\") pod \"0ebe245d-800f-49e8-aa24-05960b155f65\" (UID: \"0ebe245d-800f-49e8-aa24-05960b155f65\") " Jul 09 14:49:38.970733 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.970696 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebe245d-800f-49e8-aa24-05960b155f65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ebe245d-800f-49e8-aa24-05960b155f65" (UID: "0ebe245d-800f-49e8-aa24-05960b155f65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:49:38.972514 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.972487 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebe245d-800f-49e8-aa24-05960b155f65-kube-api-access-v9hhq" (OuterVolumeSpecName: "kube-api-access-v9hhq") pod "0ebe245d-800f-49e8-aa24-05960b155f65" (UID: "0ebe245d-800f-49e8-aa24-05960b155f65"). InnerVolumeSpecName "kube-api-access-v9hhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:49:38.972633 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:38.972534 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ebe245d-800f-49e8-aa24-05960b155f65" (UID: "0ebe245d-800f-49e8-aa24-05960b155f65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:49:39.072113 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.072075 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9hhq\" (UniqueName: \"kubernetes.io/projected/0ebe245d-800f-49e8-aa24-05960b155f65-kube-api-access-v9hhq\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:39.072113 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.072104 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebe245d-800f-49e8-aa24-05960b155f65-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:39.072113 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.072115 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ebe245d-800f-49e8-aa24-05960b155f65-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:39.364656 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.361613 2563 generic.go:356] "Generic (PLEG): container finished" podID="0ebe245d-800f-49e8-aa24-05960b155f65" containerID="0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7" exitCode=0 Jul 09 14:49:39.364656 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.362482 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" Jul 09 14:49:39.368753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.368680 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" event={"ID":"0ebe245d-800f-49e8-aa24-05960b155f65","Type":"ContainerDied","Data":"0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7"} Jul 09 14:49:39.369010 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.368945 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58" event={"ID":"0ebe245d-800f-49e8-aa24-05960b155f65","Type":"ContainerDied","Data":"ab7e1d7c71ceb6ad2cd7a1b286ce385510a051d639d89c8a4263618b4388ecee"} Jul 09 14:49:39.369125 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.369019 2563 scope.go:117] "RemoveContainer" containerID="0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7" Jul 09 14:49:39.378501 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.378485 2563 scope.go:117] "RemoveContainer" containerID="0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7" Jul 09 14:49:39.378794 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:49:39.378775 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7\": container with ID starting with 0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7 not found: ID does not exist" containerID="0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7" Jul 09 14:49:39.378852 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.378803 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7"} err="failed to get container status \"0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7\": rpc error: code = NotFound desc = could not find container \"0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7\": container with ID starting with 0aa616f44a0e30569b81730c7f1ba8b6ebb53c3d52d8c9f39a9a456257e4feb7 not found: ID does not exist" Jul 09 14:49:39.406251 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.406225 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58"] Jul 09 14:49:39.412968 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.412948 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3abd8947-5b8d7f54bc-4mq58"] Jul 09 14:49:39.892963 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.892932 2563 patch_prober.go:28] interesting pod/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 14:49:39.893129 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:39.892984 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 14:49:40.575139 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:40.575106 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ebe245d-800f-49e8-aa24-05960b155f65" path="/var/lib/kubelet/pods/0ebe245d-800f-49e8-aa24-05960b155f65/volumes" Jul 09 14:49:40.893591 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:40.893512 2563 patch_prober.go:28] interesting pod/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 14:49:40.893591 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:40.893564 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 14:49:41.893823 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:41.893789 2563 patch_prober.go:28] interesting pod/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 14:49:41.894184 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:41.893838 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 14:49:42.897272 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:42.897243 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:42.897672 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:42.897503 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:42.898165 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:42.898146 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:45.958782 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:45.958749 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j"] Jul 09 14:49:45.959222 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:45.958982 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" containerID="cri-o://629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd" gracePeriod=30 Jul 09 14:49:51.219334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.219312 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:51.367187 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367160 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-certs\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367191 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-socket\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367210 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw2rf\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-kube-api-access-hw2rf\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367230 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-podinfo\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367244 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-credential-socket\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367305 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-data\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367341 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367333 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-envoy\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367360 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-token\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367383 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764e6fc-0fff-4b29-bfb8-7161354233aa-istiod-ca-cert\") pod \"2764e6fc-0fff-4b29-bfb8-7161354233aa\" (UID: \"2764e6fc-0fff-4b29-bfb8-7161354233aa\") " Jul 09 14:49:51.367683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367575 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:51.367683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367589 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:51.367956 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367734 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.367956 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367774 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.367956 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367735 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:51.367956 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.367925 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2764e6fc-0fff-4b29-bfb8-7161354233aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:49:51.368351 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.368321 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:51.369418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.369388 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-kube-api-access-hw2rf" (OuterVolumeSpecName: "kube-api-access-hw2rf") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "kube-api-access-hw2rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:49:51.369738 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.369437 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:49:51.369928 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.369905 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:49:51.370039 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.370020 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "2764e6fc-0fff-4b29-bfb8-7161354233aa" (UID: "2764e6fc-0fff-4b29-bfb8-7161354233aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:49:51.429008 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.428981 2563 generic.go:356] "Generic (PLEG): container finished" podID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerID="629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd" exitCode=0 Jul 09 14:49:51.429104 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.429021 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" event={"ID":"2764e6fc-0fff-4b29-bfb8-7161354233aa","Type":"ContainerDied","Data":"629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd"} Jul 09 14:49:51.429104 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.429033 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" Jul 09 14:49:51.429104 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.429084 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j" event={"ID":"2764e6fc-0fff-4b29-bfb8-7161354233aa","Type":"ContainerDied","Data":"d1da0318cc7eb101c9e5c85aca365d78bccaa2687ccd47e755e2f679dc595d3f"} Jul 09 14:49:51.429235 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.429106 2563 scope.go:117] "RemoveContainer" containerID="629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd" Jul 09 14:49:51.437907 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.437892 2563 scope.go:117] "RemoveContainer" containerID="c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755" Jul 09 14:49:51.445390 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.445373 2563 scope.go:117] "RemoveContainer" containerID="629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd" Jul 09 14:49:51.445669 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:49:51.445647 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd\": container with ID starting with 629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd not found: ID does not exist" containerID="629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd" Jul 09 14:49:51.445749 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.445675 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd"} err="failed to get container status \"629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd\": rpc error: code = NotFound desc = could not find container \"629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd\": container with ID starting with 629f77e21da57dcae6cf1c13b442b5094e9d06c8c641477bebf9b262526febcd not found: ID does not exist" Jul 09 14:49:51.445749 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.445691 2563 scope.go:117] "RemoveContainer" containerID="c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755" Jul 09 14:49:51.445921 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:49:51.445906 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755\": container with ID starting with c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755 not found: ID does not exist" containerID="c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755" Jul 09 14:49:51.445961 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.445925 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755"} err="failed to get container status \"c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755\": rpc error: code = NotFound desc = could not find container \"c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755\": container with ID starting with c94891175c81ba48cdf1d66dfdda786b47879e396a30cc09ac09b1bd38d84755 not found: ID does not exist" Jul 09 14:49:51.456972 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.456949 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j"] Jul 09 14:49:51.466224 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.466204 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3abd8947-openshift-default-f98bb8955-ckb9j"] Jul 09 14:49:51.468635 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.468604 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2764e6fc-0fff-4b29-bfb8-7161354233aa-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.468635 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.468633 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.468759 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.468643 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw2rf\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-kube-api-access-hw2rf\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.468759 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.468651 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.468759 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.468660 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.468759 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.468667 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:51.468759 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:51.468674 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2764e6fc-0fff-4b29-bfb8-7161354233aa-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:49:52.574681 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:49:52.574644 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" path="/var/lib/kubelet/pods/2764e6fc-0fff-4b29-bfb8-7161354233aa/volumes" Jul 09 14:50:07.949411 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949382 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v"] Jul 09 14:50:07.949815 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949746 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" Jul 09 14:50:07.949815 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949758 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" Jul 09 14:50:07.949815 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949782 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ebe245d-800f-49e8-aa24-05960b155f65" containerName="maas-api" Jul 09 14:50:07.949815 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949791 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ebe245d-800f-49e8-aa24-05960b155f65" containerName="maas-api" Jul 09 14:50:07.949947 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949854 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ebe245d-800f-49e8-aa24-05960b155f65" containerName="maas-api" Jul 09 14:50:07.949947 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949862 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" Jul 09 14:50:07.949947 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.949872 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" Jul 09 14:50:07.953257 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.953235 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.980018 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.979999 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5788dd92-openshift-default-dockercfg-g6rs9\"" Jul 09 14:50:07.989798 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.989776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885901a-9d92-45a2-b593-38ca9907f5fb-istiod-ca-cert\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.989901 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.989831 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-certs\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.989957 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.989905 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-socket\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.990000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.989941 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnp6k\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-kube-api-access-lnp6k\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.990000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.989990 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-podinfo\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.990079 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.990011 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-data\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.990079 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.990032 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-envoy\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.990079 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.990055 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-token\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.990170 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.990128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-credential-socket\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:07.992924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:07.992902 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v"] Jul 09 14:50:08.090942 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.090914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-credential-socket\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.090960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885901a-9d92-45a2-b593-38ca9907f5fb-istiod-ca-cert\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-certs\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-socket\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091053 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnp6k\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-kube-api-access-lnp6k\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091081 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-podinfo\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091108 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-data\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091135 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-envoy\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-token\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091381 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-credential-socket\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.091543 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091419 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-certs\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.092151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091673 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-data\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.092151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091790 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885901a-9d92-45a2-b593-38ca9907f5fb-istiod-ca-cert\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.092151 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.091813 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-socket\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.093483 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.093465 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-envoy\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.094055 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.094033 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-podinfo\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.106427 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.106408 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnp6k\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-kube-api-access-lnp6k\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.116669 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.116650 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-token\") pod \"e2e-mt-5788dd92-openshift-default-686686855-5559v\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.263796 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.263771 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:08.532677 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.532655 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v"] Jul 09 14:50:08.533789 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:50:08.533765 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd885901a_9d92_45a2_b593_38ca9907f5fb.slice/crio-cab361c1302bebad51b2a5c563be0c527d766657bec975b68e591f3c0619c2b7 WatchSource:0}: Error finding container cab361c1302bebad51b2a5c563be0c527d766657bec975b68e591f3c0619c2b7: Status 404 returned error can't find the container with id cab361c1302bebad51b2a5c563be0c527d766657bec975b68e591f3c0619c2b7 Jul 09 14:50:08.538244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.538214 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:50:08.538345 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.538274 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:50:08.538345 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:08.538309 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:50:09.532855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:09.532815 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" event={"ID":"d885901a-9d92-45a2-b593-38ca9907f5fb","Type":"ContainerStarted","Data":"638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1"} Jul 09 14:50:09.533263 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:09.532876 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" event={"ID":"d885901a-9d92-45a2-b593-38ca9907f5fb","Type":"ContainerStarted","Data":"cab361c1302bebad51b2a5c563be0c527d766657bec975b68e591f3c0619c2b7"} Jul 09 14:50:09.570913 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:09.570852 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" podStartSLOduration=2.5708337329999997 podStartE2EDuration="2.570833733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:09.566758243 +0000 UTC m=+1773.587841544" watchObservedRunningTime="2026-07-09 14:50:09.570833733 +0000 UTC m=+1773.591916998" Jul 09 14:50:10.264994 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:10.264960 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:10.269793 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:10.269769 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:10.538944 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:10.538869 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:10.539801 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:10.539775 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:11.715264 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.715233 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv"] Jul 09 14:50:11.715749 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.715731 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" Jul 09 14:50:11.715827 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.715751 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2764e6fc-0fff-4b29-bfb8-7161354233aa" containerName="istio-proxy" Jul 09 14:50:11.718891 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.718870 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.723020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.722999 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5788dd92-serving-cert\"" Jul 09 14:50:11.723210 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.723194 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:50:11.731829 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.731808 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv"] Jul 09 14:50:11.821343 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.821311 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-openshift-service-ca\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.821517 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.821375 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-maas-api-tls\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.821517 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.821399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wpsl\" (UniqueName: \"kubernetes.io/projected/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-kube-api-access-6wpsl\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.922185 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.922157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-openshift-service-ca\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.922337 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.922206 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-maas-api-tls\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.922337 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.922223 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wpsl\" (UniqueName: \"kubernetes.io/projected/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-kube-api-access-6wpsl\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.922882 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.922850 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-openshift-service-ca\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.924720 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.924699 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-maas-api-tls\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:11.941379 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:11.941352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wpsl\" (UniqueName: \"kubernetes.io/projected/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-kube-api-access-6wpsl\") pod \"maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:12.029947 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:12.029926 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:12.170378 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:12.170348 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv"] Jul 09 14:50:12.170771 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:50:12.170748 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef19e0a3_72ac_496c_8fe1_e03445c8eb63.slice/crio-683181c5077ac2a56ac5479b0ce35776e85fed45c8146528a055ef56a5da3055 WatchSource:0}: Error finding container 683181c5077ac2a56ac5479b0ce35776e85fed45c8146528a055ef56a5da3055: Status 404 returned error can't find the container with id 683181c5077ac2a56ac5479b0ce35776e85fed45c8146528a055ef56a5da3055 Jul 09 14:50:12.172305 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:12.172284 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:50:12.550953 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:12.550919 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" event={"ID":"ef19e0a3-72ac-496c-8fe1-e03445c8eb63","Type":"ContainerStarted","Data":"683181c5077ac2a56ac5479b0ce35776e85fed45c8146528a055ef56a5da3055"} Jul 09 14:50:13.560459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:13.560429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" event={"ID":"ef19e0a3-72ac-496c-8fe1-e03445c8eb63","Type":"ContainerStarted","Data":"da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada"} Jul 09 14:50:13.560856 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:13.560548 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:13.609074 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:13.609022 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" podStartSLOduration=2.04039478 podStartE2EDuration="2.6090028s" totalImagesPullingTime="568.60802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:50:13.605431123 +0000 UTC m=+1777.626514419" watchObservedRunningTime="2026-07-09 14:50:13.6090028 +0000 UTC m=+1777.630086064" Jul 09 14:50:19.572181 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:19.572131 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:26.580757 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:26.580725 2563 patch_prober.go:28] interesting pod/e2e-mt-5788dd92-openshift-default-686686855-5559v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 14:50:26.581118 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:26.580771 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 14:50:27.633559 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:27.633526 2563 generic.go:356] "Generic (PLEG): container finished" podID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerID="638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1" exitCode=0 Jul 09 14:50:27.634020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:27.633603 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" event={"ID":"d885901a-9d92-45a2-b593-38ca9907f5fb","Type":"ContainerDied","Data":"638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1"} Jul 09 14:50:27.634020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:27.633973 2563 scope.go:117] "RemoveContainer" containerID="638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1" Jul 09 14:50:27.634949 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:27.634924 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:50:27.635062 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:27.635009 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:50:27.635062 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:27.635041 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:50:28.640457 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:28.640366 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" event={"ID":"d885901a-9d92-45a2-b593-38ca9907f5fb","Type":"ContainerStarted","Data":"4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443"} Jul 09 14:50:29.265044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:29.265003 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:29.265244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:29.265124 2563 patch_prober.go:28] interesting pod/e2e-mt-5788dd92-openshift-default-686686855-5559v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 09 14:50:29.265244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:29.265176 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 09 14:50:30.264853 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:30.264814 2563 patch_prober.go:28] interesting pod/e2e-mt-5788dd92-openshift-default-686686855-5559v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 09 14:50:30.265331 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:30.264874 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 09 14:50:31.264716 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:31.264679 2563 patch_prober.go:28] interesting pod/e2e-mt-5788dd92-openshift-default-686686855-5559v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 09 14:50:31.265010 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:31.264980 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 09 14:50:32.284310 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:32.284271 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:32.284756 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:32.284740 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:32.287295 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:32.287274 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:50:36.595180 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:36.595153 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:50:36.596873 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:36.596853 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:50:36.604249 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:36.604227 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:50:36.605950 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:36.605933 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:50:54.464302 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.464266 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv"] Jul 09 14:50:54.464786 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.464505 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" podUID="ef19e0a3-72ac-496c-8fe1-e03445c8eb63" containerName="maas-api" containerID="cri-o://da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada" gracePeriod=30 Jul 09 14:50:54.565803 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.565770 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" podUID="ef19e0a3-72ac-496c-8fe1-e03445c8eb63" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.56:8443/health\": dial tcp 10.132.0.56:8443: connect: connection refused" Jul 09 14:50:54.716434 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.716381 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:54.790663 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.790612 2563 generic.go:356] "Generic (PLEG): container finished" podID="ef19e0a3-72ac-496c-8fe1-e03445c8eb63" containerID="da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada" exitCode=0 Jul 09 14:50:54.790814 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.790668 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" Jul 09 14:50:54.790814 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.790717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" event={"ID":"ef19e0a3-72ac-496c-8fe1-e03445c8eb63","Type":"ContainerDied","Data":"da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada"} Jul 09 14:50:54.790814 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.790760 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv" event={"ID":"ef19e0a3-72ac-496c-8fe1-e03445c8eb63","Type":"ContainerDied","Data":"683181c5077ac2a56ac5479b0ce35776e85fed45c8146528a055ef56a5da3055"} Jul 09 14:50:54.790814 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.790781 2563 scope.go:117] "RemoveContainer" containerID="da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada" Jul 09 14:50:54.798710 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.798694 2563 scope.go:117] "RemoveContainer" containerID="da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada" Jul 09 14:50:54.798927 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:50:54.798912 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada\": container with ID starting with da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada not found: ID does not exist" containerID="da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada" Jul 09 14:50:54.798982 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.798935 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada"} err="failed to get container status \"da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada\": rpc error: code = NotFound desc = could not find container \"da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada\": container with ID starting with da77e4acdea4cc24a1eb3f6e98e7b8d91ceeb4d4626c355a3becdb8f7dc74ada not found: ID does not exist" Jul 09 14:50:54.876255 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.876234 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wpsl\" (UniqueName: \"kubernetes.io/projected/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-kube-api-access-6wpsl\") pod \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " Jul 09 14:50:54.876365 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.876313 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-maas-api-tls\") pod \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " Jul 09 14:50:54.876365 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.876342 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-openshift-service-ca\") pod \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\" (UID: \"ef19e0a3-72ac-496c-8fe1-e03445c8eb63\") " Jul 09 14:50:54.876714 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.876694 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef19e0a3-72ac-496c-8fe1-e03445c8eb63" (UID: "ef19e0a3-72ac-496c-8fe1-e03445c8eb63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:50:54.878100 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.878073 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-kube-api-access-6wpsl" (OuterVolumeSpecName: "kube-api-access-6wpsl") pod "ef19e0a3-72ac-496c-8fe1-e03445c8eb63" (UID: "ef19e0a3-72ac-496c-8fe1-e03445c8eb63"). InnerVolumeSpecName "kube-api-access-6wpsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:50:54.878181 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.878156 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef19e0a3-72ac-496c-8fe1-e03445c8eb63" (UID: "ef19e0a3-72ac-496c-8fe1-e03445c8eb63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:50:54.977244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.977189 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:50:54.977244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.977211 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:50:54.977244 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:54.977222 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wpsl\" (UniqueName: \"kubernetes.io/projected/ef19e0a3-72ac-496c-8fe1-e03445c8eb63-kube-api-access-6wpsl\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:50:55.122025 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:55.121999 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv"] Jul 09 14:50:55.125682 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:55.125663 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5788dd92-78bc4bcb54-k6ppv"] Jul 09 14:50:56.574358 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:50:56.574326 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef19e0a3-72ac-496c-8fe1-e03445c8eb63" path="/var/lib/kubelet/pods/ef19e0a3-72ac-496c-8fe1-e03445c8eb63/volumes" Jul 09 14:51:52.779810 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:52.779773 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v"] Jul 09 14:51:52.780270 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:52.779961 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" containerID="cri-o://4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443" gracePeriod=30 Jul 09 14:51:58.024612 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.024590 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:51:58.111019 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.110943 2563 generic.go:356] "Generic (PLEG): container finished" podID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerID="4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443" exitCode=0 Jul 09 14:51:58.111019 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.111003 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" Jul 09 14:51:58.111197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.111023 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" event={"ID":"d885901a-9d92-45a2-b593-38ca9907f5fb","Type":"ContainerDied","Data":"4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443"} Jul 09 14:51:58.111197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.111097 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v" event={"ID":"d885901a-9d92-45a2-b593-38ca9907f5fb","Type":"ContainerDied","Data":"cab361c1302bebad51b2a5c563be0c527d766657bec975b68e591f3c0619c2b7"} Jul 09 14:51:58.111197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.111120 2563 scope.go:117] "RemoveContainer" containerID="4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443" Jul 09 14:51:58.119821 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.119804 2563 scope.go:117] "RemoveContainer" containerID="638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1" Jul 09 14:51:58.127909 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.127891 2563 scope.go:117] "RemoveContainer" containerID="4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443" Jul 09 14:51:58.128159 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:51:58.128141 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443\": container with ID starting with 4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443 not found: ID does not exist" containerID="4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443" Jul 09 14:51:58.128219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.128168 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443"} err="failed to get container status \"4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443\": rpc error: code = NotFound desc = could not find container \"4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443\": container with ID starting with 4c0e443b703c7d1dc626ec60bf9128e86255d7fe6ad2187d43b8dec849acb443 not found: ID does not exist" Jul 09 14:51:58.128219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.128184 2563 scope.go:117] "RemoveContainer" containerID="638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1" Jul 09 14:51:58.128424 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:51:58.128404 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1\": container with ID starting with 638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1 not found: ID does not exist" containerID="638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1" Jul 09 14:51:58.128468 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.128432 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1"} err="failed to get container status \"638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1\": rpc error: code = NotFound desc = could not find container \"638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1\": container with ID starting with 638e764940fbd648b5e8e896473e6e913d4de838854d4992ab4e0e24151c7af1 not found: ID does not exist" Jul 09 14:51:58.166865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.166840 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-envoy\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.166977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.166875 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-credential-socket\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.166977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.166897 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-socket\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.166977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.166948 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885901a-9d92-45a2-b593-38ca9907f5fb-istiod-ca-cert\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.167112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.166976 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-podinfo\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.167112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167000 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-token\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.167112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167031 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnp6k\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-kube-api-access-lnp6k\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.167112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167072 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-certs\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.167313 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167117 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-data\") pod \"d885901a-9d92-45a2-b593-38ca9907f5fb\" (UID: \"d885901a-9d92-45a2-b593-38ca9907f5fb\") " Jul 09 14:51:58.167313 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167150 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:51:58.167313 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167254 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:51:58.167520 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167387 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:51:58.167520 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167413 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d885901a-9d92-45a2-b593-38ca9907f5fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:51:58.167520 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167451 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.167520 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.167468 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.168030 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.168006 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:51:58.169161 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.169130 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:51:58.169266 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.169205 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:51:58.169266 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.169219 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:51:58.169266 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.169223 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-kube-api-access-lnp6k" (OuterVolumeSpecName: "kube-api-access-lnp6k") pod "d885901a-9d92-45a2-b593-38ca9907f5fb" (UID: "d885901a-9d92-45a2-b593-38ca9907f5fb"). InnerVolumeSpecName "kube-api-access-lnp6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:51:58.268112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.268088 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.268112 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.268109 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d885901a-9d92-45a2-b593-38ca9907f5fb-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.268254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.268118 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.268254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.268128 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.268254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.268136 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnp6k\" (UniqueName: \"kubernetes.io/projected/d885901a-9d92-45a2-b593-38ca9907f5fb-kube-api-access-lnp6k\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.268254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.268145 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.268254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.268153 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d885901a-9d92-45a2-b593-38ca9907f5fb-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:51:58.449067 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.449043 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v"] Jul 09 14:51:58.453281 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.453259 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5788dd92-openshift-default-686686855-5559v"] Jul 09 14:51:58.575651 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:51:58.575611 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" path="/var/lib/kubelet/pods/d885901a-9d92-45a2-b593-38ca9907f5fb/volumes" Jul 09 14:52:56.463171 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463141 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg"] Jul 09 14:52:56.463564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463454 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" Jul 09 14:52:56.463564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463464 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" Jul 09 14:52:56.463564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463487 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef19e0a3-72ac-496c-8fe1-e03445c8eb63" containerName="maas-api" Jul 09 14:52:56.463564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463493 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef19e0a3-72ac-496c-8fe1-e03445c8eb63" containerName="maas-api" Jul 09 14:52:56.463564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463546 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef19e0a3-72ac-496c-8fe1-e03445c8eb63" containerName="maas-api" Jul 09 14:52:56.463564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463554 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" Jul 09 14:52:56.463564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.463560 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" Jul 09 14:52:56.466473 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.466454 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.469440 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.469421 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bf99b0-openshift-default-dockercfg-jj7zg\"" Jul 09 14:52:56.499664 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499645 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-envoy\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.499766 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499678 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-token\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.499766 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-credential-socket\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.499766 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499728 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-podinfo\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.499766 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499754 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-socket\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.499907 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499822 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chslw\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-kube-api-access-chslw\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.499907 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499870 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643ff6f5-902a-4c0c-9bd4-33c24664094a-istiod-ca-cert\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.499907 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-certs\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.500016 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.499912 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-data\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.515688 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.515665 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg"] Jul 09 14:52:56.600742 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643ff6f5-902a-4c0c-9bd4-33c24664094a-istiod-ca-cert\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.600862 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-certs\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.600862 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-data\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.600862 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-envoy\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.600862 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-token\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.600862 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-credential-socket\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.600862 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-podinfo\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.601250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600887 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-socket\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.601250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.600926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chslw\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-kube-api-access-chslw\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.601250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.601193 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-data\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.601406 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.601268 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-certs\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.601406 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.601322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-credential-socket\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.601406 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.601389 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-socket\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.601700 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.601680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643ff6f5-902a-4c0c-9bd4-33c24664094a-istiod-ca-cert\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.602999 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.602982 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-envoy\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.603195 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.603177 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-podinfo\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.614547 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.614528 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chslw\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-kube-api-access-chslw\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.615454 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.615439 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-token\") pod \"e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.776338 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.776312 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:56.920403 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.920372 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg"] Jul 09 14:52:56.922000 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:52:56.921970 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643ff6f5_902a_4c0c_9bd4_33c24664094a.slice/crio-0912dcc86ed4788635bf9016570d4992840b88bca21b2518405c256831397810 WatchSource:0}: Error finding container 0912dcc86ed4788635bf9016570d4992840b88bca21b2518405c256831397810: Status 404 returned error can't find the container with id 0912dcc86ed4788635bf9016570d4992840b88bca21b2518405c256831397810 Jul 09 14:52:56.924003 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.923971 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:52:56.924105 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.924045 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:52:56.924105 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:56.924090 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:52:57.421381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:57.421349 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" event={"ID":"643ff6f5-902a-4c0c-9bd4-33c24664094a","Type":"ContainerStarted","Data":"b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5"} Jul 09 14:52:57.421530 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:57.421401 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" event={"ID":"643ff6f5-902a-4c0c-9bd4-33c24664094a","Type":"ContainerStarted","Data":"0912dcc86ed4788635bf9016570d4992840b88bca21b2518405c256831397810"} Jul 09 14:52:57.462942 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:57.462897 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" podStartSLOduration=1.462880691 podStartE2EDuration="1.462880691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:52:57.460567889 +0000 UTC m=+1941.481651152" watchObservedRunningTime="2026-07-09 14:52:57.462880691 +0000 UTC m=+1941.483963955" Jul 09 14:52:57.776763 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:57.776739 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:57.781221 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:57.781202 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:58.426936 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:58.426911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:58.427772 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:58.427753 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:52:59.694400 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.694368 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh"] Jul 09 14:52:59.694772 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.694721 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" Jul 09 14:52:59.694772 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.694732 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d885901a-9d92-45a2-b593-38ca9907f5fb" containerName="istio-proxy" Jul 09 14:52:59.697799 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.697782 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.701644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.701598 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bf99b0-serving-cert\"" Jul 09 14:52:59.703026 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.703011 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:52:59.718164 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.718143 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh"] Jul 09 14:52:59.825853 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.825820 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.826002 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.825903 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82db191-ba33-479b-8681-a63449eb2f50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.826002 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.825929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpq5\" (UniqueName: \"kubernetes.io/projected/e82db191-ba33-479b-8681-a63449eb2f50-kube-api-access-5fpq5\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.926421 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.926394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpq5\" (UniqueName: \"kubernetes.io/projected/e82db191-ba33-479b-8681-a63449eb2f50-kube-api-access-5fpq5\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.926560 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.926447 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.926560 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.926493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82db191-ba33-479b-8681-a63449eb2f50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.926707 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:52:59.926645 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-serving-cert: secret "maas-api-e2e-inf-a-bf99b0-serving-cert" not found Jul 09 14:52:59.926769 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:52:59.926722 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls podName:e82db191-ba33-479b-8681-a63449eb2f50 nodeName:}" failed. No retries permitted until 2026-07-09 14:53:00.426696631 +0000 UTC m=+1944.447779889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls") pod "maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" (UID: "e82db191-ba33-479b-8681-a63449eb2f50") : secret "maas-api-e2e-inf-a-bf99b0-serving-cert" not found Jul 09 14:52:59.927105 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.927083 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82db191-ba33-479b-8681-a63449eb2f50-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:52:59.950732 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:52:59.950673 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpq5\" (UniqueName: \"kubernetes.io/projected/e82db191-ba33-479b-8681-a63449eb2f50-kube-api-access-5fpq5\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:53:00.430314 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:00.430289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:53:00.432496 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:00.432473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:53:00.608248 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:00.608217 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:53:00.998209 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:00.998184 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh"] Jul 09 14:53:01.000107 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:53:01.000079 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82db191_ba33_479b_8681_a63449eb2f50.slice/crio-81de113e08ae4ac3eb8314c3bcec8149c61d40d73ad3af87c412aff53e4fc55a WatchSource:0}: Error finding container 81de113e08ae4ac3eb8314c3bcec8149c61d40d73ad3af87c412aff53e4fc55a: Status 404 returned error can't find the container with id 81de113e08ae4ac3eb8314c3bcec8149c61d40d73ad3af87c412aff53e4fc55a Jul 09 14:53:01.442354 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:01.442316 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" event={"ID":"e82db191-ba33-479b-8681-a63449eb2f50","Type":"ContainerStarted","Data":"81de113e08ae4ac3eb8314c3bcec8149c61d40d73ad3af87c412aff53e4fc55a"} Jul 09 14:53:02.448903 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:02.448867 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" event={"ID":"e82db191-ba33-479b-8681-a63449eb2f50","Type":"ContainerStarted","Data":"e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae"} Jul 09 14:53:02.449336 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:02.448986 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:53:02.487264 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:02.487219 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" podStartSLOduration=2.84331575 podStartE2EDuration="3.487206787s" totalImagesPullingTime="643.891037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:53:02.48391317 +0000 UTC m=+1946.504996434" watchObservedRunningTime="2026-07-09 14:53:02.487206787 +0000 UTC m=+1946.508290051" Jul 09 14:53:04.760029 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.759998 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz"] Jul 09 14:53:04.763713 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.763688 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.767484 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.767464 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ab1ad0-openshift-default-dockercfg-62552\"" Jul 09 14:53:04.800500 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.800480 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz"] Jul 09 14:53:04.864743 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.864707 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-credential-socket\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.865067 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.865046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-certs\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.865220 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.865206 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-envoy\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.865356 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.865344 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-podinfo\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.865479 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.865467 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-socket\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.865637 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.865608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-token\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.865785 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.865769 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vtl\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-kube-api-access-m9vtl\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.865912 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.865899 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-data\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.866020 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.866009 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istiod-ca-cert\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.966750 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.966712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-certs\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.966750 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.966754 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-envoy\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.966967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.966776 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-podinfo\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.966967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.966798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-socket\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.966967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.966837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-token\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.966967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.966882 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vtl\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-kube-api-access-m9vtl\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967178 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-data\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967178 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967057 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istiod-ca-cert\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967178 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967087 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-credential-socket\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967350 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967171 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-certs\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967350 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967215 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-socket\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967453 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-data\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967506 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967461 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-credential-socket\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.967758 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.967741 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istiod-ca-cert\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.969044 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.969024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-envoy\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.969321 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.969305 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-podinfo\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.980172 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.980149 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vtl\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-kube-api-access-m9vtl\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:04.981465 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:04.981443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-token\") pod \"e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:05.073296 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.073228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:05.240740 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.240701 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz"] Jul 09 14:53:05.249140 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:53:05.249114 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc32aba8_b48e_4376_a8ac_f8a42da2d0f9.slice/crio-6c43975fff6112f5054b076b97bb176402994877a5d625cbed4c3052b68e0bed WatchSource:0}: Error finding container 6c43975fff6112f5054b076b97bb176402994877a5d625cbed4c3052b68e0bed: Status 404 returned error can't find the container with id 6c43975fff6112f5054b076b97bb176402994877a5d625cbed4c3052b68e0bed Jul 09 14:53:05.251198 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.251163 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:05.251288 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.251225 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:05.251288 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.251250 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:05.465971 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.465878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" event={"ID":"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9","Type":"ContainerStarted","Data":"55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f"} Jul 09 14:53:05.466114 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.465969 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" event={"ID":"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9","Type":"ContainerStarted","Data":"6c43975fff6112f5054b076b97bb176402994877a5d625cbed4c3052b68e0bed"} Jul 09 14:53:05.502147 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:05.502102 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" podStartSLOduration=1.502086453 podStartE2EDuration="1.502086453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:53:05.50121242 +0000 UTC m=+1949.522295683" watchObservedRunningTime="2026-07-09 14:53:05.502086453 +0000 UTC m=+1949.523169717" Jul 09 14:53:06.073747 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:06.073717 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:06.078265 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:06.078244 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:06.470974 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:06.470897 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:06.471823 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:06.471805 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:07.671564 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.671536 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9"] Jul 09 14:53:07.674942 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.674925 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.680262 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.680243 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ab1ad0-serving-cert\"" Jul 09 14:53:07.691552 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.691528 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9"] Jul 09 14:53:07.793547 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.793504 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532c9812-88f2-42d6-aa05-06f5f668270a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.793741 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.793656 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krpf7\" (UniqueName: \"kubernetes.io/projected/532c9812-88f2-42d6-aa05-06f5f668270a-kube-api-access-krpf7\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.793741 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.793686 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.894976 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.894948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532c9812-88f2-42d6-aa05-06f5f668270a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.895122 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.894996 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krpf7\" (UniqueName: \"kubernetes.io/projected/532c9812-88f2-42d6-aa05-06f5f668270a-kube-api-access-krpf7\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.895122 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.895017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.895122 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:53:07.895115 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-serving-cert: secret "maas-api-e2e-inf-b-ab1ad0-serving-cert" not found Jul 09 14:53:07.895225 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:53:07.895175 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls podName:532c9812-88f2-42d6-aa05-06f5f668270a nodeName:}" failed. No retries permitted until 2026-07-09 14:53:08.395159127 +0000 UTC m=+1952.416242369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls") pod "maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" (UID: "532c9812-88f2-42d6-aa05-06f5f668270a") : secret "maas-api-e2e-inf-b-ab1ad0-serving-cert" not found Jul 09 14:53:07.895662 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.895643 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532c9812-88f2-42d6-aa05-06f5f668270a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:07.913338 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:07.913316 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krpf7\" (UniqueName: \"kubernetes.io/projected/532c9812-88f2-42d6-aa05-06f5f668270a-kube-api-access-krpf7\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:08.399551 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:08.399517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:08.401865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:08.401847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:08.460832 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:08.460804 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:53:08.586254 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:08.586227 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:08.743036 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:08.743008 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9"] Jul 09 14:53:08.745632 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:53:08.745594 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532c9812_88f2_42d6_aa05_06f5f668270a.slice/crio-e90c14db78b4e0ccbba717bf4beb6bca3056781f887b1e73a3bf9228fd3ab2ad WatchSource:0}: Error finding container e90c14db78b4e0ccbba717bf4beb6bca3056781f887b1e73a3bf9228fd3ab2ad: Status 404 returned error can't find the container with id e90c14db78b4e0ccbba717bf4beb6bca3056781f887b1e73a3bf9228fd3ab2ad Jul 09 14:53:09.488512 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:09.488415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" event={"ID":"532c9812-88f2-42d6-aa05-06f5f668270a","Type":"ContainerStarted","Data":"8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1"} Jul 09 14:53:09.488512 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:09.488473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" event={"ID":"532c9812-88f2-42d6-aa05-06f5f668270a","Type":"ContainerStarted","Data":"e90c14db78b4e0ccbba717bf4beb6bca3056781f887b1e73a3bf9228fd3ab2ad"} Jul 09 14:53:09.488512 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:09.488502 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:09.531939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:09.531892 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" podStartSLOduration=2.227715389 podStartE2EDuration="2.531871432s" totalImagesPullingTime="304.156043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:53:09.531452109 +0000 UTC m=+1953.552535397" watchObservedRunningTime="2026-07-09 14:53:09.531871432 +0000 UTC m=+1953.552954709" Jul 09 14:53:14.467789 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:14.467743 2563 patch_prober.go:28] interesting pod/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 14:53:14.468264 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:14.467823 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 14:53:15.500310 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:15.500281 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:53:16.527114 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:16.527081 2563 generic.go:356] "Generic (PLEG): container finished" podID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerID="b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5" exitCode=0 Jul 09 14:53:16.527490 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:16.527155 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" event={"ID":"643ff6f5-902a-4c0c-9bd4-33c24664094a","Type":"ContainerDied","Data":"b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5"} Jul 09 14:53:16.527539 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:16.527527 2563 scope.go:117] "RemoveContainer" containerID="b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5" Jul 09 14:53:16.528558 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:16.528533 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:16.528641 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:16.528611 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:16.528683 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:16.528674 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:17.538795 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:17.538757 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" event={"ID":"643ff6f5-902a-4c0c-9bd4-33c24664094a","Type":"ContainerStarted","Data":"1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6"} Jul 09 14:53:17.777820 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:17.777779 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:53:17.778015 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:17.777909 2563 patch_prober.go:28] interesting pod/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:53:17.778015 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:17.777953 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:53:18.777319 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:18.777273 2563 patch_prober.go:28] interesting pod/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:53:18.777858 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:18.777340 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:53:19.777472 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:19.777435 2563 patch_prober.go:28] interesting pod/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:53:19.777895 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:19.777500 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:53:20.777335 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:20.777299 2563 patch_prober.go:28] interesting pod/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 09 14:53:20.777520 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:20.777362 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 09 14:53:21.780714 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:21.780684 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:53:21.781176 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:21.781067 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:53:21.781642 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:21.781606 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:53:22.511773 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.511730 2563 patch_prober.go:28] interesting pod/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 14:53:22.512829 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.511828 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 14:53:22.575886 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.575859 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerID="55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f" exitCode=0 Jul 09 14:53:22.575997 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.575926 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" event={"ID":"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9","Type":"ContainerDied","Data":"55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f"} Jul 09 14:53:22.576448 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.576431 2563 scope.go:117] "RemoveContainer" containerID="55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f" Jul 09 14:53:22.577302 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.577278 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:22.577360 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.577342 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:22.577402 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:22.577374 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:53:23.583658 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:23.583611 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" event={"ID":"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9","Type":"ContainerStarted","Data":"56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572"} Jul 09 14:53:24.074393 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:24.074357 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:24.074682 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:24.074659 2563 patch_prober.go:28] interesting pod/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 14:53:24.074732 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:24.074718 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 14:53:25.074040 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:25.074005 2563 patch_prober.go:28] interesting pod/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 14:53:25.074428 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:25.074058 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 14:53:26.074109 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:26.074075 2563 patch_prober.go:28] interesting pod/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 14:53:26.074494 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:26.074127 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 14:53:27.073508 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:27.073476 2563 patch_prober.go:28] interesting pod/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 09 14:53:27.073677 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:27.073524 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 09 14:53:28.079762 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:28.079730 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:28.080150 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:28.080111 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:53:28.080980 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:53:28.080960 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:54:13.074993 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.074957 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh"] Jul 09 14:54:13.075514 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.075172 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" podUID="e82db191-ba33-479b-8681-a63449eb2f50" containerName="maas-api" containerID="cri-o://e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae" gracePeriod=30 Jul 09 14:54:13.319596 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.319566 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:54:13.456240 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.456182 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82db191-ba33-479b-8681-a63449eb2f50-openshift-service-ca\") pod \"e82db191-ba33-479b-8681-a63449eb2f50\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " Jul 09 14:54:13.456352 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.456267 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls\") pod \"e82db191-ba33-479b-8681-a63449eb2f50\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " Jul 09 14:54:13.456352 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.456306 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fpq5\" (UniqueName: \"kubernetes.io/projected/e82db191-ba33-479b-8681-a63449eb2f50-kube-api-access-5fpq5\") pod \"e82db191-ba33-479b-8681-a63449eb2f50\" (UID: \"e82db191-ba33-479b-8681-a63449eb2f50\") " Jul 09 14:54:13.456522 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.456502 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82db191-ba33-479b-8681-a63449eb2f50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e82db191-ba33-479b-8681-a63449eb2f50" (UID: "e82db191-ba33-479b-8681-a63449eb2f50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:54:13.458263 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.458236 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82db191-ba33-479b-8681-a63449eb2f50-kube-api-access-5fpq5" (OuterVolumeSpecName: "kube-api-access-5fpq5") pod "e82db191-ba33-479b-8681-a63449eb2f50" (UID: "e82db191-ba33-479b-8681-a63449eb2f50"). InnerVolumeSpecName "kube-api-access-5fpq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:54:13.458356 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.458281 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e82db191-ba33-479b-8681-a63449eb2f50" (UID: "e82db191-ba33-479b-8681-a63449eb2f50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:54:13.557872 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.557850 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fpq5\" (UniqueName: \"kubernetes.io/projected/e82db191-ba33-479b-8681-a63449eb2f50-kube-api-access-5fpq5\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:54:13.557872 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.557870 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82db191-ba33-479b-8681-a63449eb2f50-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:54:13.557990 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.557879 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e82db191-ba33-479b-8681-a63449eb2f50-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:54:13.850483 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.850447 2563 generic.go:356] "Generic (PLEG): container finished" podID="e82db191-ba33-479b-8681-a63449eb2f50" containerID="e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae" exitCode=0 Jul 09 14:54:13.850601 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.850497 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" Jul 09 14:54:13.850601 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.850533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" event={"ID":"e82db191-ba33-479b-8681-a63449eb2f50","Type":"ContainerDied","Data":"e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae"} Jul 09 14:54:13.850601 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.850595 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh" event={"ID":"e82db191-ba33-479b-8681-a63449eb2f50","Type":"ContainerDied","Data":"81de113e08ae4ac3eb8314c3bcec8149c61d40d73ad3af87c412aff53e4fc55a"} Jul 09 14:54:13.850731 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.850648 2563 scope.go:117] "RemoveContainer" containerID="e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae" Jul 09 14:54:13.861079 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.861053 2563 scope.go:117] "RemoveContainer" containerID="e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae" Jul 09 14:54:13.861367 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:54:13.861346 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae\": container with ID starting with e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae not found: ID does not exist" containerID="e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae" Jul 09 14:54:13.861424 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.861377 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae"} err="failed to get container status \"e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae\": rpc error: code = NotFound desc = could not find container \"e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae\": container with ID starting with e8d0938a8436594aced5b445e1a805114d9a96d86362bb7ce6a716dbe540fbae not found: ID does not exist" Jul 09 14:54:13.878905 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.878877 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh"] Jul 09 14:54:13.887532 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:13.887502 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf99b0-5b8b88d87c-z6rfh"] Jul 09 14:54:14.575221 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:14.575188 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82db191-ba33-479b-8681-a63449eb2f50" path="/var/lib/kubelet/pods/e82db191-ba33-479b-8681-a63449eb2f50/volumes" Jul 09 14:54:55.058235 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:55.058201 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg"] Jul 09 14:54:55.058752 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:55.058447 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" containerID="cri-o://1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6" gracePeriod=30 Jul 09 14:54:56.222338 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.222306 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9"] Jul 09 14:54:56.222784 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.222493 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" podUID="532c9812-88f2-42d6-aa05-06f5f668270a" containerName="maas-api" containerID="cri-o://8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1" gracePeriod=30 Jul 09 14:54:56.473560 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.473497 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:54:56.482550 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.482528 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krpf7\" (UniqueName: \"kubernetes.io/projected/532c9812-88f2-42d6-aa05-06f5f668270a-kube-api-access-krpf7\") pod \"532c9812-88f2-42d6-aa05-06f5f668270a\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " Jul 09 14:54:56.482679 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.482560 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532c9812-88f2-42d6-aa05-06f5f668270a-openshift-service-ca\") pod \"532c9812-88f2-42d6-aa05-06f5f668270a\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " Jul 09 14:54:56.482679 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.482578 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls\") pod \"532c9812-88f2-42d6-aa05-06f5f668270a\" (UID: \"532c9812-88f2-42d6-aa05-06f5f668270a\") " Jul 09 14:54:56.482962 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.482942 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532c9812-88f2-42d6-aa05-06f5f668270a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "532c9812-88f2-42d6-aa05-06f5f668270a" (UID: "532c9812-88f2-42d6-aa05-06f5f668270a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:54:56.484555 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.484530 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "532c9812-88f2-42d6-aa05-06f5f668270a" (UID: "532c9812-88f2-42d6-aa05-06f5f668270a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:54:56.484555 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.484540 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532c9812-88f2-42d6-aa05-06f5f668270a-kube-api-access-krpf7" (OuterVolumeSpecName: "kube-api-access-krpf7") pod "532c9812-88f2-42d6-aa05-06f5f668270a" (UID: "532c9812-88f2-42d6-aa05-06f5f668270a"). InnerVolumeSpecName "kube-api-access-krpf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:54:56.583527 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.583497 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532c9812-88f2-42d6-aa05-06f5f668270a-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:54:56.583527 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.583523 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532c9812-88f2-42d6-aa05-06f5f668270a-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:54:56.583707 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:56.583608 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krpf7\" (UniqueName: \"kubernetes.io/projected/532c9812-88f2-42d6-aa05-06f5f668270a-kube-api-access-krpf7\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:54:57.094915 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.094881 2563 generic.go:356] "Generic (PLEG): container finished" podID="532c9812-88f2-42d6-aa05-06f5f668270a" containerID="8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1" exitCode=0 Jul 09 14:54:57.095074 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.094942 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" Jul 09 14:54:57.095074 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.094979 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" event={"ID":"532c9812-88f2-42d6-aa05-06f5f668270a","Type":"ContainerDied","Data":"8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1"} Jul 09 14:54:57.095074 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.095069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9" event={"ID":"532c9812-88f2-42d6-aa05-06f5f668270a","Type":"ContainerDied","Data":"e90c14db78b4e0ccbba717bf4beb6bca3056781f887b1e73a3bf9228fd3ab2ad"} Jul 09 14:54:57.095250 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.095100 2563 scope.go:117] "RemoveContainer" containerID="8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1" Jul 09 14:54:57.103839 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.103824 2563 scope.go:117] "RemoveContainer" containerID="8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1" Jul 09 14:54:57.104105 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:54:57.104077 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1\": container with ID starting with 8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1 not found: ID does not exist" containerID="8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1" Jul 09 14:54:57.104199 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.104103 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1"} err="failed to get container status \"8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1\": rpc error: code = NotFound desc = could not find container \"8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1\": container with ID starting with 8ad95a391a61afebcda76f938d17e87ccdacc4259b1baf266de6775f2d3b19b1 not found: ID does not exist" Jul 09 14:54:57.118684 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.118660 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9"] Jul 09 14:54:57.124866 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:57.124847 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab1ad0-dbdd447-rmdm9"] Jul 09 14:54:58.575246 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:54:58.575165 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="532c9812-88f2-42d6-aa05-06f5f668270a" path="/var/lib/kubelet/pods/532c9812-88f2-42d6-aa05-06f5f668270a/volumes" Jul 09 14:55:00.302107 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.302086 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:55:00.415227 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415171 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-envoy\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415227 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415202 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643ff6f5-902a-4c0c-9bd4-33c24664094a-istiod-ca-cert\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415230 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-podinfo\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415255 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-certs\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415288 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chslw\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-kube-api-access-chslw\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415381 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415334 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-token\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415587 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415397 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-credential-socket\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415587 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415436 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-data\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415587 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415463 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-socket\") pod \"643ff6f5-902a-4c0c-9bd4-33c24664094a\" (UID: \"643ff6f5-902a-4c0c-9bd4-33c24664094a\") " Jul 09 14:55:00.415777 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415651 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:00.415777 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415671 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:00.415887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415770 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.415887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415818 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.415887 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415859 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:00.416041 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.415955 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643ff6f5-902a-4c0c-9bd4-33c24664094a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:55:00.416353 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.416333 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-data" (OuterVolumeSpecName: "istio-data") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:00.417349 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.417325 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:55:00.417442 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.417333 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:00.417785 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.417767 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-kube-api-access-chslw" (OuterVolumeSpecName: "kube-api-access-chslw") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "kube-api-access-chslw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:00.417851 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.417785 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-token" (OuterVolumeSpecName: "istio-token") pod "643ff6f5-902a-4c0c-9bd4-33c24664094a" (UID: "643ff6f5-902a-4c0c-9bd4-33c24664094a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:00.516585 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.516563 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.516585 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.516584 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chslw\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-kube-api-access-chslw\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.516721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.516593 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.516721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.516602 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.516721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.516609 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.516721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.516636 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643ff6f5-902a-4c0c-9bd4-33c24664094a-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:00.516721 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:00.516645 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643ff6f5-902a-4c0c-9bd4-33c24664094a-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:01.117837 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.117808 2563 generic.go:356] "Generic (PLEG): container finished" podID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerID="1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6" exitCode=0 Jul 09 14:55:01.117983 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.117874 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" Jul 09 14:55:01.117983 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.117902 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" event={"ID":"643ff6f5-902a-4c0c-9bd4-33c24664094a","Type":"ContainerDied","Data":"1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6"} Jul 09 14:55:01.117983 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.117974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg" event={"ID":"643ff6f5-902a-4c0c-9bd4-33c24664094a","Type":"ContainerDied","Data":"0912dcc86ed4788635bf9016570d4992840b88bca21b2518405c256831397810"} Jul 09 14:55:01.118141 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.118010 2563 scope.go:117] "RemoveContainer" containerID="1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6" Jul 09 14:55:01.128239 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.128219 2563 scope.go:117] "RemoveContainer" containerID="b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5" Jul 09 14:55:01.136729 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.136708 2563 scope.go:117] "RemoveContainer" containerID="1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6" Jul 09 14:55:01.137127 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:01.137099 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6\": container with ID starting with 1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6 not found: ID does not exist" containerID="1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6" Jul 09 14:55:01.137225 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.137135 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6"} err="failed to get container status \"1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6\": rpc error: code = NotFound desc = could not find container \"1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6\": container with ID starting with 1ec4ff6ecf85426a37905cf42ae0080dc09ad5f7a7c13ccdc393e84f2056cbf6 not found: ID does not exist" Jul 09 14:55:01.137225 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.137161 2563 scope.go:117] "RemoveContainer" containerID="b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5" Jul 09 14:55:01.137527 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:01.137506 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5\": container with ID starting with b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5 not found: ID does not exist" containerID="b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5" Jul 09 14:55:01.137631 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.137536 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5"} err="failed to get container status \"b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5\": rpc error: code = NotFound desc = could not find container \"b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5\": container with ID starting with b33e7b01da61c4cdf863e97c79af4bddaca54e8bf8f77598d9d8cc1053ecd2e5 not found: ID does not exist" Jul 09 14:55:01.140038 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.140017 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg"] Jul 09 14:55:01.151236 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:01.151215 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf99b0-openshift-default-767b65586f-zsvwg"] Jul 09 14:55:02.434757 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:02.434725 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz"] Jul 09 14:55:02.435182 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:02.434970 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" containerID="cri-o://56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572" gracePeriod=30 Jul 09 14:55:02.575843 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:02.575809 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" path="/var/lib/kubelet/pods/643ff6f5-902a-4c0c-9bd4-33c24664094a/volumes" Jul 09 14:55:03.350881 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.350848 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s"] Jul 09 14:55:03.351185 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351173 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" Jul 09 14:55:03.351185 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351185 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351205 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82db191-ba33-479b-8681-a63449eb2f50" containerName="maas-api" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351211 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82db191-ba33-479b-8681-a63449eb2f50" containerName="maas-api" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351220 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="532c9812-88f2-42d6-aa05-06f5f668270a" containerName="maas-api" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351228 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="532c9812-88f2-42d6-aa05-06f5f668270a" containerName="maas-api" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351281 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="532c9812-88f2-42d6-aa05-06f5f668270a" containerName="maas-api" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351290 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351296 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" Jul 09 14:55:03.351359 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.351301 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82db191-ba33-479b-8681-a63449eb2f50" containerName="maas-api" Jul 09 14:55:03.355848 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.355829 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.359157 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.359135 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-966126-openshift-default-dockercfg-bk2q4\"" Jul 09 14:55:03.372311 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.372290 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s"] Jul 09 14:55:03.440742 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.440710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbwc\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-kube-api-access-lzbwc\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.440798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-data\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.440843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-credential-socket\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.440881 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-socket\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.440902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-certs\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.440933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463446d-33c2-4afd-bd69-6be662499004-istiod-ca-cert\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.440980 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463446d-33c2-4afd-bd69-6be662499004-istio-podinfo\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.441002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-envoy\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.441162 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.441073 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-istio-token\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.541598 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-istio-token\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.541745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbwc\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-kube-api-access-lzbwc\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.541745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-data\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.541745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-credential-socket\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.541745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541699 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-socket\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.541745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-certs\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.541745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541744 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463446d-33c2-4afd-bd69-6be662499004-istiod-ca-cert\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.542050 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463446d-33c2-4afd-bd69-6be662499004-istio-podinfo\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.542050 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.541813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-envoy\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.542158 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.542083 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-data\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.542158 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.542123 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-credential-socket\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.542253 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.542186 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-certs\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.542329 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.542308 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-socket\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.542534 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.542513 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463446d-33c2-4afd-bd69-6be662499004-istiod-ca-cert\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.544184 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.544161 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-envoy\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.544387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.544369 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463446d-33c2-4afd-bd69-6be662499004-istio-podinfo\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.551363 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.551343 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-istio-token\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.552159 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.552137 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbwc\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-kube-api-access-lzbwc\") pod \"e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.665928 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.665874 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:03.808418 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.808390 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s"] Jul 09 14:55:03.810708 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:03.810683 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0463446d_33c2_4afd_bd69_6be662499004.slice/crio-520613e5c3a824ebc530c4e0fad2057a5e85b785c2cff7d50ae02abcbce8722a WatchSource:0}: Error finding container 520613e5c3a824ebc530c4e0fad2057a5e85b785c2cff7d50ae02abcbce8722a: Status 404 returned error can't find the container with id 520613e5c3a824ebc530c4e0fad2057a5e85b785c2cff7d50ae02abcbce8722a Jul 09 14:55:03.813278 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.813251 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:03.813355 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.813311 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:03.813355 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:03.813338 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:04.140709 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:04.140675 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" event={"ID":"0463446d-33c2-4afd-bd69-6be662499004","Type":"ContainerStarted","Data":"4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621"} Jul 09 14:55:04.140860 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:04.140731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" event={"ID":"0463446d-33c2-4afd-bd69-6be662499004","Type":"ContainerStarted","Data":"520613e5c3a824ebc530c4e0fad2057a5e85b785c2cff7d50ae02abcbce8722a"} Jul 09 14:55:04.164738 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:04.164695 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" podStartSLOduration=1.164679432 podStartE2EDuration="1.164679432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:04.162085132 +0000 UTC m=+2068.183168415" watchObservedRunningTime="2026-07-09 14:55:04.164679432 +0000 UTC m=+2068.185762695" Jul 09 14:55:04.667009 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:04.666978 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:04.671530 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:04.671504 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:05.146697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:05.146668 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:05.147528 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:05.147509 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:06.226823 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.226793 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7"] Jul 09 14:55:06.227327 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.227308 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" Jul 09 14:55:06.227407 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.227330 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="643ff6f5-902a-4c0c-9bd4-33c24664094a" containerName="istio-proxy" Jul 09 14:55:06.230322 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.230301 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.233326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.233308 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-966126-serving-cert\"" Jul 09 14:55:06.233432 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.233330 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:55:06.243137 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.243112 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7"] Jul 09 14:55:06.263094 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.263068 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.263203 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.263167 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.263266 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.263213 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7gpl\" (UniqueName: \"kubernetes.io/projected/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-kube-api-access-j7gpl\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.364560 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.364530 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.364735 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.364570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7gpl\" (UniqueName: \"kubernetes.io/projected/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-kube-api-access-j7gpl\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.364735 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.364610 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.364735 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:06.364697 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-966126-serving-cert: secret "maas-api-e2e-api-a-966126-serving-cert" not found Jul 09 14:55:06.364912 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:06.364760 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls podName:6a9e775a-6c2f-4393-86bb-e1e9b44d07d4 nodeName:}" failed. No retries permitted until 2026-07-09 14:55:06.864740063 +0000 UTC m=+2070.885823323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls") pod "maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" (UID: "6a9e775a-6c2f-4393-86bb-e1e9b44d07d4") : secret "maas-api-e2e-api-a-966126-serving-cert" not found Jul 09 14:55:06.365221 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.365204 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.374187 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.374170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7gpl\" (UniqueName: \"kubernetes.io/projected/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-kube-api-access-j7gpl\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.869643 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.869602 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:06.871907 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:06.871881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls\") pod \"maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:07.142370 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.142295 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:07.287934 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.287907 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7"] Jul 09 14:55:07.289048 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:07.289022 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9e775a_6c2f_4393_86bb_e1e9b44d07d4.slice/crio-593a1579d3c4ecb3925433a2cde10161b69ddb859155dfdfb007d3f091f08633 WatchSource:0}: Error finding container 593a1579d3c4ecb3925433a2cde10161b69ddb859155dfdfb007d3f091f08633: Status 404 returned error can't find the container with id 593a1579d3c4ecb3925433a2cde10161b69ddb859155dfdfb007d3f091f08633 Jul 09 14:55:07.688610 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.688583 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:55:07.777233 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777207 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-data\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777244 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-credential-socket\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777263 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9vtl\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-kube-api-access-m9vtl\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777280 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-socket\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777295 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istiod-ca-cert\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777347 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-envoy\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777362 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-podinfo\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777387 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777386 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-certs\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777433 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-token\") pod \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\" (UID: \"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9\") " Jul 09 14:55:07.777924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777513 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:07.777924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777769 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:07.777924 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.777797 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.778139 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.778068 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:55:07.778139 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.778124 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:07.778316 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.778293 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:07.779355 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.779325 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-kube-api-access-m9vtl" (OuterVolumeSpecName: "kube-api-access-m9vtl") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "kube-api-access-m9vtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:07.779455 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.779416 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:55:07.779811 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.779790 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:07.779866 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.779819 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" (UID: "fc32aba8-b48e-4376-a8ac-f8a42da2d0f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:07.879093 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879068 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.879093 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879091 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.879223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879099 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.879223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879107 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.879223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879114 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.879223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879122 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vtl\" (UniqueName: \"kubernetes.io/projected/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-kube-api-access-m9vtl\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.879223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879129 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:07.879223 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:07.879138 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:08.165588 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.165509 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerID="56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572" exitCode=0 Jul 09 14:55:08.165588 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.165575 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" Jul 09 14:55:08.165804 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.165588 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" event={"ID":"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9","Type":"ContainerDied","Data":"56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572"} Jul 09 14:55:08.165804 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.165656 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz" event={"ID":"fc32aba8-b48e-4376-a8ac-f8a42da2d0f9","Type":"ContainerDied","Data":"6c43975fff6112f5054b076b97bb176402994877a5d625cbed4c3052b68e0bed"} Jul 09 14:55:08.165804 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.165687 2563 scope.go:117] "RemoveContainer" containerID="56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572" Jul 09 14:55:08.167405 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.167382 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" event={"ID":"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4","Type":"ContainerStarted","Data":"330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396"} Jul 09 14:55:08.167520 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.167431 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" event={"ID":"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4","Type":"ContainerStarted","Data":"593a1579d3c4ecb3925433a2cde10161b69ddb859155dfdfb007d3f091f08633"} Jul 09 14:55:08.167520 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.167461 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:08.175157 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.175142 2563 scope.go:117] "RemoveContainer" containerID="55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f" Jul 09 14:55:08.182990 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.182972 2563 scope.go:117] "RemoveContainer" containerID="56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572" Jul 09 14:55:08.183261 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:08.183244 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572\": container with ID starting with 56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572 not found: ID does not exist" containerID="56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572" Jul 09 14:55:08.183308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.183273 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572"} err="failed to get container status \"56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572\": rpc error: code = NotFound desc = could not find container \"56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572\": container with ID starting with 56c1221f6d3d685790290d9a6a68dd53a34ba6662fea0ff3d48010ccb4b49572 not found: ID does not exist" Jul 09 14:55:08.183308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.183289 2563 scope.go:117] "RemoveContainer" containerID="55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f" Jul 09 14:55:08.183555 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:08.183529 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f\": container with ID starting with 55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f not found: ID does not exist" containerID="55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f" Jul 09 14:55:08.183598 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.183564 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f"} err="failed to get container status \"55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f\": rpc error: code = NotFound desc = could not find container \"55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f\": container with ID starting with 55a8afc7a1e4a6f45719778555a87dba0600de4cd8d0908096e16bede2e0a71f not found: ID does not exist" Jul 09 14:55:08.192062 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.192025 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" podStartSLOduration=1.657255782 podStartE2EDuration="2.192013067s" totalImagesPullingTime="534.757285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:08.191073088 +0000 UTC m=+2072.212156376" watchObservedRunningTime="2026-07-09 14:55:08.192013067 +0000 UTC m=+2072.213096329" Jul 09 14:55:08.228290 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.228264 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz"] Jul 09 14:55:08.243788 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.243766 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ab1ad0-openshift-default-6c477f4877-cnmlz"] Jul 09 14:55:08.574595 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:08.574568 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" path="/var/lib/kubelet/pods/fc32aba8-b48e-4376-a8ac-f8a42da2d0f9/volumes" Jul 09 14:55:11.431739 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.431709 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj"] Jul 09 14:55:11.432093 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.432082 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" Jul 09 14:55:11.432093 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.432093 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" Jul 09 14:55:11.432164 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.432111 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" Jul 09 14:55:11.432164 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.432116 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" Jul 09 14:55:11.432232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.432180 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" Jul 09 14:55:11.432232 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.432190 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc32aba8-b48e-4376-a8ac-f8a42da2d0f9" containerName="istio-proxy" Jul 09 14:55:11.435183 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.435164 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.438293 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.438273 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-11ff2c-openshift-default-dockercfg-nkf9f\"" Jul 09 14:55:11.454060 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.454036 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj"] Jul 09 14:55:11.508610 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-data\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508637 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-envoy\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508726 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hs7d\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-kube-api-access-4hs7d\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508799 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917fba45-e526-4a02-b51a-f84575eb0013-istio-podinfo\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508799 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508763 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-socket\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-certs\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508825 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-istio-token\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508865 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508847 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917fba45-e526-4a02-b51a-f84575eb0013-istiod-ca-cert\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.508969 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.508916 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-credential-socket\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.609706 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917fba45-e526-4a02-b51a-f84575eb0013-istio-podinfo\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.609847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-socket\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.609847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-certs\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.609847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-istio-token\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.609847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917fba45-e526-4a02-b51a-f84575eb0013-istiod-ca-cert\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610029 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609868 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-credential-socket\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610029 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-data\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610029 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.609958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-envoy\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610139 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.610102 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-socket\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.610158 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-certs\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.610197 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-credential-socket\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610275 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.610243 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hs7d\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-kube-api-access-4hs7d\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610608 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.610588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-data\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.610782 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.610762 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917fba45-e526-4a02-b51a-f84575eb0013-istiod-ca-cert\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.612230 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.612214 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-envoy\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.612358 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.612344 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917fba45-e526-4a02-b51a-f84575eb0013-istio-podinfo\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.620035 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.620011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-istio-token\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.621077 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.621058 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hs7d\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-kube-api-access-4hs7d\") pod \"e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.746299 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.746245 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:11.885169 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.885147 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj"] Jul 09 14:55:11.887516 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:11.887488 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917fba45_e526_4a02_b51a_f84575eb0013.slice/crio-77cd4003494f60d77425d36f4d7c33f54cbaf599a8d9fea06301fabe96292835 WatchSource:0}: Error finding container 77cd4003494f60d77425d36f4d7c33f54cbaf599a8d9fea06301fabe96292835: Status 404 returned error can't find the container with id 77cd4003494f60d77425d36f4d7c33f54cbaf599a8d9fea06301fabe96292835 Jul 09 14:55:11.889710 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.889682 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:11.889792 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.889744 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:11.889792 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:11.889774 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:12.191820 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:12.191785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" event={"ID":"917fba45-e526-4a02-b51a-f84575eb0013","Type":"ContainerStarted","Data":"a89fa7713e22e28e566502655a75e8023f64556d01a7ba816ecc4617afd53d65"} Jul 09 14:55:12.191976 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:12.191838 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" event={"ID":"917fba45-e526-4a02-b51a-f84575eb0013","Type":"ContainerStarted","Data":"77cd4003494f60d77425d36f4d7c33f54cbaf599a8d9fea06301fabe96292835"} Jul 09 14:55:12.214347 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:12.214298 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" podStartSLOduration=1.214282279 podStartE2EDuration="1.214282279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:12.212829461 +0000 UTC m=+2076.233912746" watchObservedRunningTime="2026-07-09 14:55:12.214282279 +0000 UTC m=+2076.235365542" Jul 09 14:55:12.747351 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:12.747323 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:12.752831 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:12.752809 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:13.197566 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:13.197529 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:13.198538 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:13.198516 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:14.178905 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.178877 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:14.294231 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.294197 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj"] Jul 09 14:55:14.297771 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.297753 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.302468 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.302450 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-11ff2c-serving-cert\"" Jul 09 14:55:14.314868 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.314841 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj"] Jul 09 14:55:14.433983 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.433893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfks9\" (UniqueName: \"kubernetes.io/projected/bf890c82-621d-4ac3-8965-3ae8eee331eb-kube-api-access-kfks9\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.433983 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.433937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf890c82-621d-4ac3-8965-3ae8eee331eb-maas-api-tls\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.434195 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.434025 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf890c82-621d-4ac3-8965-3ae8eee331eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.534948 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.534920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf890c82-621d-4ac3-8965-3ae8eee331eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.535087 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.534977 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfks9\" (UniqueName: \"kubernetes.io/projected/bf890c82-621d-4ac3-8965-3ae8eee331eb-kube-api-access-kfks9\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.535130 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.535080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf890c82-621d-4ac3-8965-3ae8eee331eb-maas-api-tls\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.535560 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.535542 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf890c82-621d-4ac3-8965-3ae8eee331eb-openshift-service-ca\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.537382 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.537363 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf890c82-621d-4ac3-8965-3ae8eee331eb-maas-api-tls\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.549589 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.549566 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfks9\" (UniqueName: \"kubernetes.io/projected/bf890c82-621d-4ac3-8965-3ae8eee331eb-kube-api-access-kfks9\") pod \"maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.613362 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.613339 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:14.742470 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.742364 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj"] Jul 09 14:55:14.745097 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:14.745068 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf890c82_621d_4ac3_8965_3ae8eee331eb.slice/crio-10d556909e48bba05647685c28bec929617615fc5c764a24f7f3a2728c25ead6 WatchSource:0}: Error finding container 10d556909e48bba05647685c28bec929617615fc5c764a24f7f3a2728c25ead6: Status 404 returned error can't find the container with id 10d556909e48bba05647685c28bec929617615fc5c764a24f7f3a2728c25ead6 Jul 09 14:55:14.746481 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:14.746461 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 14:55:15.209665 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:15.209606 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" event={"ID":"bf890c82-621d-4ac3-8965-3ae8eee331eb","Type":"ContainerStarted","Data":"7024582dfa5ec9c5b7ed046cdcd47aeb739b50fa0642fad9691290e3d8a4e42c"} Jul 09 14:55:15.210049 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:15.209683 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:15.210049 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:15.209699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" event={"ID":"bf890c82-621d-4ac3-8965-3ae8eee331eb","Type":"ContainerStarted","Data":"10d556909e48bba05647685c28bec929617615fc5c764a24f7f3a2728c25ead6"} Jul 09 14:55:15.231127 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:15.231082 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" podStartSLOduration=0.901955332 podStartE2EDuration="1.231066344s" totalImagesPullingTime="329.111012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:15.229058591 +0000 UTC m=+2079.250141850" watchObservedRunningTime="2026-07-09 14:55:15.231066344 +0000 UTC m=+2079.252149633" Jul 09 14:55:21.188736 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:21.188705 2563 patch_prober.go:28] interesting pod/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 14:55:21.189100 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:21.188754 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 14:55:21.220403 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:21.220380 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:22.248247 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:22.248158 2563 generic.go:356] "Generic (PLEG): container finished" podID="0463446d-33c2-4afd-bd69-6be662499004" containerID="4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621" exitCode=0 Jul 09 14:55:22.248247 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:22.248231 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" event={"ID":"0463446d-33c2-4afd-bd69-6be662499004","Type":"ContainerDied","Data":"4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621"} Jul 09 14:55:22.248745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:22.248595 2563 scope.go:117] "RemoveContainer" containerID="4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621" Jul 09 14:55:22.249518 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:22.249486 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:22.249640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:22.249550 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:22.249640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:22.249594 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:23.260007 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:23.259966 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" event={"ID":"0463446d-33c2-4afd-bd69-6be662499004","Type":"ContainerStarted","Data":"7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436"} Jul 09 14:55:23.666892 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:23.666787 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:23.667083 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:23.666920 2563 patch_prober.go:28] interesting pod/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 09 14:55:23.667083 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:23.666962 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 09 14:55:24.667222 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:24.667186 2563 patch_prober.go:28] interesting pod/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 09 14:55:24.667714 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:24.667246 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 09 14:55:25.666451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:25.666413 2563 patch_prober.go:28] interesting pod/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 09 14:55:25.666794 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:25.666764 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 09 14:55:26.667296 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:26.667248 2563 patch_prober.go:28] interesting pod/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 09 14:55:26.667805 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:26.667350 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 09 14:55:27.686202 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:27.686171 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:27.686666 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:27.686458 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:27.687281 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:27.687261 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:27.795375 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:27.795339 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7"] Jul 09 14:55:27.795727 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:27.795695 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" podUID="6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" containerName="maas-api" containerID="cri-o://330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396" gracePeriod=30 Jul 09 14:55:27.847927 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:27.847895 2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9e775a_6c2f_4393_86bb_e1e9b44d07d4.slice/crio-conmon-330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9e775a_6c2f_4393_86bb_e1e9b44d07d4.slice/crio-330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396.scope\": RecentStats: unable to find data in memory cache]" Jul 09 14:55:28.048425 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.048400 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:28.165459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.165427 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls\") pod \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " Jul 09 14:55:28.165644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.165505 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7gpl\" (UniqueName: \"kubernetes.io/projected/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-kube-api-access-j7gpl\") pod \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " Jul 09 14:55:28.165644 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.165635 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-openshift-service-ca\") pod \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\" (UID: \"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4\") " Jul 09 14:55:28.165976 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.165955 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" (UID: "6a9e775a-6c2f-4393-86bb-e1e9b44d07d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:55:28.167573 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.167548 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" (UID: "6a9e775a-6c2f-4393-86bb-e1e9b44d07d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:55:28.167705 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.167668 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-kube-api-access-j7gpl" (OuterVolumeSpecName: "kube-api-access-j7gpl") pod "6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" (UID: "6a9e775a-6c2f-4393-86bb-e1e9b44d07d4"). InnerVolumeSpecName "kube-api-access-j7gpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:28.266933 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.266907 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:28.267013 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.266937 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:28.267013 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.266950 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7gpl\" (UniqueName: \"kubernetes.io/projected/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4-kube-api-access-j7gpl\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:28.315388 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.315359 2563 generic.go:356] "Generic (PLEG): container finished" podID="6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" containerID="330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396" exitCode=0 Jul 09 14:55:28.315478 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.315406 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" Jul 09 14:55:28.315478 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.315446 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" event={"ID":"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4","Type":"ContainerDied","Data":"330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396"} Jul 09 14:55:28.315561 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.315500 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7" event={"ID":"6a9e775a-6c2f-4393-86bb-e1e9b44d07d4","Type":"ContainerDied","Data":"593a1579d3c4ecb3925433a2cde10161b69ddb859155dfdfb007d3f091f08633"} Jul 09 14:55:28.315561 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.315525 2563 scope.go:117] "RemoveContainer" containerID="330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396" Jul 09 14:55:28.325850 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.325830 2563 scope.go:117] "RemoveContainer" containerID="330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396" Jul 09 14:55:28.326151 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:28.326133 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396\": container with ID starting with 330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396 not found: ID does not exist" containerID="330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396" Jul 09 14:55:28.326211 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.326158 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396"} err="failed to get container status \"330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396\": rpc error: code = NotFound desc = could not find container \"330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396\": container with ID starting with 330c7fac362680884299c8279b9ee42f14d8603b87beac8fae7554e287eb1396 not found: ID does not exist" Jul 09 14:55:28.337868 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.337846 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7"] Jul 09 14:55:28.344051 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.344030 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-966126-57cfcfc55b-j5pp7"] Jul 09 14:55:28.576668 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:28.576582 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" path="/var/lib/kubelet/pods/6a9e775a-6c2f-4393-86bb-e1e9b44d07d4/volumes" Jul 09 14:55:29.239753 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:29.239722 2563 patch_prober.go:28] interesting pod/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 14:55:29.240140 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:29.239836 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 14:55:30.335208 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:30.335161 2563 generic.go:356] "Generic (PLEG): container finished" podID="917fba45-e526-4a02-b51a-f84575eb0013" containerID="a89fa7713e22e28e566502655a75e8023f64556d01a7ba816ecc4617afd53d65" exitCode=0 Jul 09 14:55:30.335940 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:30.335344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" event={"ID":"917fba45-e526-4a02-b51a-f84575eb0013","Type":"ContainerDied","Data":"a89fa7713e22e28e566502655a75e8023f64556d01a7ba816ecc4617afd53d65"} Jul 09 14:55:30.336015 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:30.335997 2563 scope.go:117] "RemoveContainer" containerID="a89fa7713e22e28e566502655a75e8023f64556d01a7ba816ecc4617afd53d65" Jul 09 14:55:30.337745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:30.337685 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:30.337880 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:30.337851 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:30.337974 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:30.337947 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:31.342847 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:31.342813 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" event={"ID":"917fba45-e526-4a02-b51a-f84575eb0013","Type":"ContainerStarted","Data":"c736e1c77fec184cfdb03de0a098b35d6e7a719907788b80b38347e806122371"} Jul 09 14:55:31.747190 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:31.746722 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:31.747190 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:31.746832 2563 patch_prober.go:28] interesting pod/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:55:31.747190 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:31.746872 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:55:32.746781 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:32.746742 2563 patch_prober.go:28] interesting pod/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:55:32.747269 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:32.746809 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:55:33.747682 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:33.747644 2563 patch_prober.go:28] interesting pod/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:55:33.748156 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:33.747707 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:55:33.861961 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:33.861912 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s"] Jul 09 14:55:33.862436 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:33.862406 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" containerID="cri-o://7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436" gracePeriod=30 Jul 09 14:55:34.747057 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:34.747025 2563 patch_prober.go:28] interesting pod/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 09 14:55:34.747326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:34.747301 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 09 14:55:35.249463 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.249432 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj"] Jul 09 14:55:35.249875 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.249674 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" podUID="bf890c82-621d-4ac3-8965-3ae8eee331eb" containerName="maas-api" containerID="cri-o://7024582dfa5ec9c5b7ed046cdcd47aeb739b50fa0642fad9691290e3d8a4e42c" gracePeriod=30 Jul 09 14:55:35.374246 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.374219 2563 generic.go:356] "Generic (PLEG): container finished" podID="bf890c82-621d-4ac3-8965-3ae8eee331eb" containerID="7024582dfa5ec9c5b7ed046cdcd47aeb739b50fa0642fad9691290e3d8a4e42c" exitCode=0 Jul 09 14:55:35.374375 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.374297 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" event={"ID":"bf890c82-621d-4ac3-8965-3ae8eee331eb","Type":"ContainerDied","Data":"7024582dfa5ec9c5b7ed046cdcd47aeb739b50fa0642fad9691290e3d8a4e42c"} Jul 09 14:55:35.498697 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.498671 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:35.632335 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.632306 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfks9\" (UniqueName: \"kubernetes.io/projected/bf890c82-621d-4ac3-8965-3ae8eee331eb-kube-api-access-kfks9\") pod \"bf890c82-621d-4ac3-8965-3ae8eee331eb\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " Jul 09 14:55:35.632493 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.632361 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf890c82-621d-4ac3-8965-3ae8eee331eb-maas-api-tls\") pod \"bf890c82-621d-4ac3-8965-3ae8eee331eb\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " Jul 09 14:55:35.632493 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.632414 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf890c82-621d-4ac3-8965-3ae8eee331eb-openshift-service-ca\") pod \"bf890c82-621d-4ac3-8965-3ae8eee331eb\" (UID: \"bf890c82-621d-4ac3-8965-3ae8eee331eb\") " Jul 09 14:55:35.632778 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.632757 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf890c82-621d-4ac3-8965-3ae8eee331eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf890c82-621d-4ac3-8965-3ae8eee331eb" (UID: "bf890c82-621d-4ac3-8965-3ae8eee331eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:55:35.634284 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.634258 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf890c82-621d-4ac3-8965-3ae8eee331eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf890c82-621d-4ac3-8965-3ae8eee331eb" (UID: "bf890c82-621d-4ac3-8965-3ae8eee331eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 14:55:35.634362 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.634302 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf890c82-621d-4ac3-8965-3ae8eee331eb-kube-api-access-kfks9" (OuterVolumeSpecName: "kube-api-access-kfks9") pod "bf890c82-621d-4ac3-8965-3ae8eee331eb" (UID: "bf890c82-621d-4ac3-8965-3ae8eee331eb"). InnerVolumeSpecName "kube-api-access-kfks9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:35.733745 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.733726 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf890c82-621d-4ac3-8965-3ae8eee331eb-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:35.733834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.733748 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf890c82-621d-4ac3-8965-3ae8eee331eb-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:35.733834 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.733766 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfks9\" (UniqueName: \"kubernetes.io/projected/bf890c82-621d-4ac3-8965-3ae8eee331eb-kube-api-access-kfks9\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:35.750255 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.750231 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:35.750613 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.750596 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:35.751184 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:35.751167 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:36.381894 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.381860 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" event={"ID":"bf890c82-621d-4ac3-8965-3ae8eee331eb","Type":"ContainerDied","Data":"10d556909e48bba05647685c28bec929617615fc5c764a24f7f3a2728c25ead6"} Jul 09 14:55:36.382308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.381952 2563 scope.go:117] "RemoveContainer" containerID="7024582dfa5ec9c5b7ed046cdcd47aeb739b50fa0642fad9691290e3d8a4e42c" Jul 09 14:55:36.382308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.381875 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj" Jul 09 14:55:36.407662 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.407639 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj"] Jul 09 14:55:36.412368 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.412343 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11ff2c-575644c66d-ks7hj"] Jul 09 14:55:36.574441 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.574419 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf890c82-621d-4ac3-8965-3ae8eee331eb" path="/var/lib/kubelet/pods/bf890c82-621d-4ac3-8965-3ae8eee331eb/volumes" Jul 09 14:55:36.632155 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.632108 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:55:36.636732 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.636711 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 14:55:36.639013 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.638994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:55:36.643221 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:36.643194 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 14:55:39.115852 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.115830 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:39.263776 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.263755 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-socket\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.263922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.263816 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463446d-33c2-4afd-bd69-6be662499004-istio-podinfo\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.263922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.263857 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbwc\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-kube-api-access-lzbwc\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.263922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.263880 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463446d-33c2-4afd-bd69-6be662499004-istiod-ca-cert\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.263922 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.263902 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-envoy\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.264197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.263955 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-credential-socket\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.264197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.263990 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-data\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.264197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264008 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:39.264197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264024 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-certs\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.264197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264115 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-istio-token\") pod \"0463446d-33c2-4afd-bd69-6be662499004\" (UID: \"0463446d-33c2-4afd-bd69-6be662499004\") " Jul 09 14:55:39.264546 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264312 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:39.264642 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264404 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.264642 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264328 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0463446d-33c2-4afd-bd69-6be662499004-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:55:39.264756 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264560 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:39.264884 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.264863 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-data" (OuterVolumeSpecName: "istio-data") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:39.266074 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.266045 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-kube-api-access-lzbwc" (OuterVolumeSpecName: "kube-api-access-lzbwc") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "kube-api-access-lzbwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:39.266255 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.266233 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-istio-token" (OuterVolumeSpecName: "istio-token") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:39.266451 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.266430 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0463446d-33c2-4afd-bd69-6be662499004-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:55:39.266502 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.266429 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0463446d-33c2-4afd-bd69-6be662499004" (UID: "0463446d-33c2-4afd-bd69-6be662499004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:39.365613 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365591 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.365717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365613 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.365717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365644 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.365717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365652 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.365717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365659 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0463446d-33c2-4afd-bd69-6be662499004-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.365717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365666 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbwc\" (UniqueName: \"kubernetes.io/projected/0463446d-33c2-4afd-bd69-6be662499004-kube-api-access-lzbwc\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.365717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365674 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0463446d-33c2-4afd-bd69-6be662499004-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.365717 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.365681 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0463446d-33c2-4afd-bd69-6be662499004-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:39.405514 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.405490 2563 generic.go:356] "Generic (PLEG): container finished" podID="0463446d-33c2-4afd-bd69-6be662499004" containerID="7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436" exitCode=0 Jul 09 14:55:39.405653 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.405545 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" Jul 09 14:55:39.405653 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.405587 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" event={"ID":"0463446d-33c2-4afd-bd69-6be662499004","Type":"ContainerDied","Data":"7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436"} Jul 09 14:55:39.405787 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.405664 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s" event={"ID":"0463446d-33c2-4afd-bd69-6be662499004","Type":"ContainerDied","Data":"520613e5c3a824ebc530c4e0fad2057a5e85b785c2cff7d50ae02abcbce8722a"} Jul 09 14:55:39.405787 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.405688 2563 scope.go:117] "RemoveContainer" containerID="7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436" Jul 09 14:55:39.414424 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.414409 2563 scope.go:117] "RemoveContainer" containerID="4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621" Jul 09 14:55:39.421970 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.421953 2563 scope.go:117] "RemoveContainer" containerID="7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436" Jul 09 14:55:39.422220 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:39.422201 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436\": container with ID starting with 7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436 not found: ID does not exist" containerID="7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436" Jul 09 14:55:39.422290 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.422232 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436"} err="failed to get container status \"7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436\": rpc error: code = NotFound desc = could not find container \"7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436\": container with ID starting with 7705f127057bf45291ff3ad4fed958721e24295d5cef6ebfd8eee339bcd7f436 not found: ID does not exist" Jul 09 14:55:39.422290 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.422255 2563 scope.go:117] "RemoveContainer" containerID="4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621" Jul 09 14:55:39.422506 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:39.422489 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621\": container with ID starting with 4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621 not found: ID does not exist" containerID="4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621" Jul 09 14:55:39.422550 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.422512 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621"} err="failed to get container status \"4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621\": rpc error: code = NotFound desc = could not find container \"4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621\": container with ID starting with 4df13a516eb3cbe93468d3df25ecc5020815ac1c48069d3d9f311895370b2621 not found: ID does not exist" Jul 09 14:55:39.429230 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.429209 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s"] Jul 09 14:55:39.433855 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:39.433837 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-966126-openshift-default-7dddb44f9-n7k7s"] Jul 09 14:55:40.575551 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:40.575517 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0463446d-33c2-4afd-bd69-6be662499004" path="/var/lib/kubelet/pods/0463446d-33c2-4afd-bd69-6be662499004/volumes" Jul 09 14:55:41.313661 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:41.313602 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj"] Jul 09 14:55:41.313873 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:41.313847 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" containerID="cri-o://c736e1c77fec184cfdb03de0a098b35d6e7a719907788b80b38347e806122371" gracePeriod=30 Jul 09 14:55:42.205680 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.205649 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t"] Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.205973 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" containerName="maas-api" Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.205984 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" containerName="maas-api" Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206005 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206011 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206017 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf890c82-621d-4ac3-8965-3ae8eee331eb" containerName="maas-api" Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206024 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf890c82-621d-4ac3-8965-3ae8eee331eb" containerName="maas-api" Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206030 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" Jul 09 14:55:42.206046 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206035 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" Jul 09 14:55:42.206306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206091 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf890c82-621d-4ac3-8965-3ae8eee331eb" containerName="maas-api" Jul 09 14:55:42.206306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206099 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" Jul 09 14:55:42.206306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206106 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0463446d-33c2-4afd-bd69-6be662499004" containerName="istio-proxy" Jul 09 14:55:42.206306 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.206113 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9e775a-6c2f-4393-86bb-e1e9b44d07d4" containerName="maas-api" Jul 09 14:55:42.209186 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.209170 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.211549 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.211530 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-0d877f-openshift-default-dockercfg-phzbt\"" Jul 09 14:55:42.221458 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.221439 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t"] Jul 09 14:55:42.388242 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388203 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-token\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388419 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-certs\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388419 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-envoy\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388419 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388351 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-socket\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388419 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388368 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-credential-socket\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388419 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388405 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-podinfo\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388466 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-data\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388503 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istiod-ca-cert\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.388640 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.388523 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjw9\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-kube-api-access-7qjw9\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489724 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489647 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-data\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489724 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istiod-ca-cert\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489724 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjw9\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-kube-api-access-7qjw9\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-token\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489785 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-certs\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-envoy\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489901 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-socket\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.489967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-credential-socket\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.490212 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.489978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-podinfo\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.490212 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.490126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-data\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.490212 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.490180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-certs\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.490376 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.490287 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-credential-socket\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.490376 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.490350 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-socket\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.490376 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.490371 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istiod-ca-cert\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.492243 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.492221 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-envoy\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.492338 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.492314 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-podinfo\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.501100 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.501073 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-token\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.502226 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.502198 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjw9\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-kube-api-access-7qjw9\") pod \"e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.518895 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.518876 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:42.660783 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.660758 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t"] Jul 09 14:55:42.661676 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:42.661650 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabab67af_a29a_4ee5_985e_3a38c90c6ee2.slice/crio-32a00186ac9afbc10eb012a723434987260c0075a45bf17ebe1cb76dd667fca5 WatchSource:0}: Error finding container 32a00186ac9afbc10eb012a723434987260c0075a45bf17ebe1cb76dd667fca5: Status 404 returned error can't find the container with id 32a00186ac9afbc10eb012a723434987260c0075a45bf17ebe1cb76dd667fca5 Jul 09 14:55:42.664454 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.664414 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:42.664537 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.664511 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:42.664586 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:42.664550 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:43.432131 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:43.432089 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" event={"ID":"abab67af-a29a-4ee5-985e-3a38c90c6ee2","Type":"ContainerStarted","Data":"2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac"} Jul 09 14:55:43.432538 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:43.432157 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" event={"ID":"abab67af-a29a-4ee5-985e-3a38c90c6ee2","Type":"ContainerStarted","Data":"32a00186ac9afbc10eb012a723434987260c0075a45bf17ebe1cb76dd667fca5"} Jul 09 14:55:43.461665 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:43.461599 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" podStartSLOduration=1.461583899 podStartE2EDuration="1.461583899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:43.458306228 +0000 UTC m=+2107.479389506" watchObservedRunningTime="2026-07-09 14:55:43.461583899 +0000 UTC m=+2107.482667163" Jul 09 14:55:43.519366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:43.519344 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:43.523901 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:43.523875 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:44.438132 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:44.438101 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:44.439305 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:44.439279 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:55:45.649778 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.649751 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n"] Jul 09 14:55:45.653291 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.653274 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.656334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.656312 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-0d877f-serving-cert\"" Jul 09 14:55:45.656334 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.656326 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-djq6x\"" Jul 09 14:55:45.669755 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.669732 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n"] Jul 09 14:55:45.814700 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.814669 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.814844 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.814705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hcd\" (UniqueName: \"kubernetes.io/projected/226842e0-dfde-4359-b361-77c62f77a5bd-kube-api-access-r5hcd\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.814844 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.814731 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226842e0-dfde-4359-b361-77c62f77a5bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.915768 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.915693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.915768 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.915728 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hcd\" (UniqueName: \"kubernetes.io/projected/226842e0-dfde-4359-b361-77c62f77a5bd-kube-api-access-r5hcd\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.915962 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.915767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226842e0-dfde-4359-b361-77c62f77a5bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.915962 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:45.915835 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-serving-cert: secret "maas-api-e2e-shared-a-0d877f-serving-cert" not found Jul 09 14:55:45.915962 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:45.915901 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls podName:226842e0-dfde-4359-b361-77c62f77a5bd nodeName:}" failed. No retries permitted until 2026-07-09 14:55:46.415885378 +0000 UTC m=+2110.436968620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls") pod "maas-api-e2e-shared-a-0d877f-569797dc69-2957n" (UID: "226842e0-dfde-4359-b361-77c62f77a5bd") : secret "maas-api-e2e-shared-a-0d877f-serving-cert" not found Jul 09 14:55:45.916358 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.916341 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226842e0-dfde-4359-b361-77c62f77a5bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:45.925420 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:45.925397 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hcd\" (UniqueName: \"kubernetes.io/projected/226842e0-dfde-4359-b361-77c62f77a5bd-kube-api-access-r5hcd\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:46.419353 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.419327 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:46.421598 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.421577 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d877f-569797dc69-2957n\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:46.449371 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.449345 2563 generic.go:356] "Generic (PLEG): container finished" podID="917fba45-e526-4a02-b51a-f84575eb0013" containerID="c736e1c77fec184cfdb03de0a098b35d6e7a719907788b80b38347e806122371" exitCode=0 Jul 09 14:55:46.449505 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.449418 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" event={"ID":"917fba45-e526-4a02-b51a-f84575eb0013","Type":"ContainerDied","Data":"c736e1c77fec184cfdb03de0a098b35d6e7a719907788b80b38347e806122371"} Jul 09 14:55:46.449505 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.449482 2563 scope.go:117] "RemoveContainer" containerID="a89fa7713e22e28e566502655a75e8023f64556d01a7ba816ecc4617afd53d65" Jul 09 14:55:46.563775 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.563750 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:46.579728 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.579704 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:46.717977 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.717953 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n"] Jul 09 14:55:46.720202 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:46.720176 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226842e0_dfde_4359_b361_77c62f77a5bd.slice/crio-7c6df130937a58bb5ce667516040c77f996f471b18df9fc30ce4cbd247f2c029 WatchSource:0}: Error finding container 7c6df130937a58bb5ce667516040c77f996f471b18df9fc30ce4cbd247f2c029: Status 404 returned error can't find the container with id 7c6df130937a58bb5ce667516040c77f996f471b18df9fc30ce4cbd247f2c029 Jul 09 14:55:46.721818 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.721796 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917fba45-e526-4a02-b51a-f84575eb0013-istio-podinfo\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.721919 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.721841 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-data\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.721919 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.721864 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-certs\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.721919 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.721893 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-envoy\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.722121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.721939 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hs7d\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-kube-api-access-4hs7d\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.722121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.721968 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-credential-socket\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.722121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.721994 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-istio-token\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.722121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722013 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917fba45-e526-4a02-b51a-f84575eb0013-istiod-ca-cert\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.722121 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722041 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-socket\") pod \"917fba45-e526-4a02-b51a-f84575eb0013\" (UID: \"917fba45-e526-4a02-b51a-f84575eb0013\") " Jul 09 14:55:46.722366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722123 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:46.722366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722306 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:46.722480 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722427 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.722480 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722444 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.722584 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722560 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:46.722663 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722608 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917fba45-e526-4a02-b51a-f84575eb0013-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 14:55:46.722663 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.722608 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-data" (OuterVolumeSpecName: "istio-data") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:46.724275 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.724217 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-istio-token" (OuterVolumeSpecName: "istio-token") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:46.724526 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.724477 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/917fba45-e526-4a02-b51a-f84575eb0013-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 14:55:46.724526 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.724483 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-kube-api-access-4hs7d" (OuterVolumeSpecName: "kube-api-access-4hs7d") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "kube-api-access-4hs7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 14:55:46.724733 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.724714 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "917fba45-e526-4a02-b51a-f84575eb0013" (UID: "917fba45-e526-4a02-b51a-f84575eb0013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 14:55:46.823189 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.823166 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917fba45-e526-4a02-b51a-f84575eb0013-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.823189 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.823188 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.823326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.823196 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.823326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.823204 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hs7d\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-kube-api-access-4hs7d\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.823326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.823213 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917fba45-e526-4a02-b51a-f84575eb0013-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.823326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.823221 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917fba45-e526-4a02-b51a-f84575eb0013-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:46.823326 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:46.823229 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917fba45-e526-4a02-b51a-f84575eb0013-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 14:55:47.456952 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.456884 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" Jul 09 14:55:47.456952 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.456909 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj" event={"ID":"917fba45-e526-4a02-b51a-f84575eb0013","Type":"ContainerDied","Data":"77cd4003494f60d77425d36f4d7c33f54cbaf599a8d9fea06301fabe96292835"} Jul 09 14:55:47.457143 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.457006 2563 scope.go:117] "RemoveContainer" containerID="c736e1c77fec184cfdb03de0a098b35d6e7a719907788b80b38347e806122371" Jul 09 14:55:47.458453 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.458431 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" event={"ID":"226842e0-dfde-4359-b361-77c62f77a5bd","Type":"ContainerStarted","Data":"04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074"} Jul 09 14:55:47.458548 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.458476 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" event={"ID":"226842e0-dfde-4359-b361-77c62f77a5bd","Type":"ContainerStarted","Data":"7c6df130937a58bb5ce667516040c77f996f471b18df9fc30ce4cbd247f2c029"} Jul 09 14:55:47.458605 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.458572 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:47.488508 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.488453 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" podStartSLOduration=2.02302066 podStartE2EDuration="2.488442118s" totalImagesPullingTime="465.421458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:47.485472899 +0000 UTC m=+2111.506556192" watchObservedRunningTime="2026-07-09 14:55:47.488442118 +0000 UTC m=+2111.509525381" Jul 09 14:55:47.501873 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.501852 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj"] Jul 09 14:55:47.508998 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:47.508975 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-11ff2c-openshift-default-7fc6899547-hxlnj"] Jul 09 14:55:48.574747 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:48.574713 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="917fba45-e526-4a02-b51a-f84575eb0013" path="/var/lib/kubelet/pods/917fba45-e526-4a02-b51a-f84575eb0013/volumes" Jul 09 14:55:50.611772 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.611742 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g"] Jul 09 14:55:50.612269 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.612248 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" Jul 09 14:55:50.612358 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.612271 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" Jul 09 14:55:50.612358 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.612295 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" Jul 09 14:55:50.612358 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.612303 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" Jul 09 14:55:50.612526 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.612409 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" Jul 09 14:55:50.612526 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.612420 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="917fba45-e526-4a02-b51a-f84575eb0013" containerName="istio-proxy" Jul 09 14:55:50.616792 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.616771 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.619570 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.619552 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d74e21-openshift-default-dockercfg-wp7lb\"" Jul 09 14:55:50.632470 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.632451 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g"] Jul 09 14:55:50.648453 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648434 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-credential-socket\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648541 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648462 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-certs\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648718 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-socket\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648846 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648727 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-data\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648846 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648751 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istiod-ca-cert\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648846 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-podinfo\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648984 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648852 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-token\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648984 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgrd\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-kube-api-access-ddgrd\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.648984 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.648905 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-envoy\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.749779 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.749748 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-socket\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.749779 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.749782 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-data\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.749972 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.749802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istiod-ca-cert\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.749972 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.749828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-podinfo\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.749972 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.749877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-token\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.749972 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.749894 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgrd\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-kube-api-access-ddgrd\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.749970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-envoy\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.750025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-credential-socket\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.750059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-certs\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.750157 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-socket\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750197 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.750187 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-data\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750459 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.750368 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-credential-socket\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750523 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.750453 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-certs\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.750523 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.750468 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istiod-ca-cert\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.752173 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.752149 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-envoy\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.752272 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.752236 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-podinfo\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.763000 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.762975 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-token\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.763207 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.763188 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgrd\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-kube-api-access-ddgrd\") pod \"e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:50.926665 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:50.926579 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:51.096177 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.096152 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g"] Jul 09 14:55:51.096627 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:51.096592 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c0bfb8_e1f7_4f2d_bbe3_b298b4272288.slice/crio-1a701d5560a0368de918812e7af69790aa19d8215bb8b112c9e7a30be239154d WatchSource:0}: Error finding container 1a701d5560a0368de918812e7af69790aa19d8215bb8b112c9e7a30be239154d: Status 404 returned error can't find the container with id 1a701d5560a0368de918812e7af69790aa19d8215bb8b112c9e7a30be239154d Jul 09 14:55:51.098711 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.098681 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:51.098781 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.098742 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:51.098781 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.098768 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:55:51.483705 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.483673 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" event={"ID":"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288","Type":"ContainerStarted","Data":"a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3"} Jul 09 14:55:51.483877 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.483724 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" event={"ID":"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288","Type":"ContainerStarted","Data":"1a701d5560a0368de918812e7af69790aa19d8215bb8b112c9e7a30be239154d"} Jul 09 14:55:51.508828 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.508757 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" podStartSLOduration=1.508738837 podStartE2EDuration="1.508738837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:51.507560055 +0000 UTC m=+2115.528643343" watchObservedRunningTime="2026-07-09 14:55:51.508738837 +0000 UTC m=+2115.529822101" Jul 09 14:55:51.927671 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:51.927641 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:52.932190 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:52.932153 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:53.474081 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.474043 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 14:55:53.498130 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.498091 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:53.499823 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.499794 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:55:53.676525 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.676490 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br"] Jul 09 14:55:53.680713 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.680687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.683357 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.683337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d74e21-serving-cert\"" Jul 09 14:55:53.710002 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.709978 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br"] Jul 09 14:55:53.773677 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.773658 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58pqq\" (UniqueName: \"kubernetes.io/projected/9b633244-0920-4211-9f0a-66aeed3b2e2a-kube-api-access-58pqq\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.773773 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.773688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b633244-0920-4211-9f0a-66aeed3b2e2a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.773813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.773785 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.874332 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.874299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.874494 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.874355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58pqq\" (UniqueName: \"kubernetes.io/projected/9b633244-0920-4211-9f0a-66aeed3b2e2a-kube-api-access-58pqq\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.874494 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.874394 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b633244-0920-4211-9f0a-66aeed3b2e2a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.874494 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:53.874457 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-serving-cert: secret "maas-api-e2e-shared-b-d74e21-serving-cert" not found Jul 09 14:55:53.874608 ip-10-0-133-55 kubenswrapper[2563]: E0709 14:55:53.874522 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls podName:9b633244-0920-4211-9f0a-66aeed3b2e2a nodeName:}" failed. No retries permitted until 2026-07-09 14:55:54.374501181 +0000 UTC m=+2118.395584436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls") pod "maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" (UID: "9b633244-0920-4211-9f0a-66aeed3b2e2a") : secret "maas-api-e2e-shared-b-d74e21-serving-cert" not found Jul 09 14:55:53.875094 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.875076 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b633244-0920-4211-9f0a-66aeed3b2e2a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:53.888168 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:53.888145 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58pqq\" (UniqueName: \"kubernetes.io/projected/9b633244-0920-4211-9f0a-66aeed3b2e2a-kube-api-access-58pqq\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:54.379283 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:54.379248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:54.381658 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:54.381607 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls\") pod \"maas-api-e2e-shared-b-d74e21-5fcd869db5-595br\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:54.591880 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:54.591848 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:54.772351 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:54.771807 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br"] Jul 09 14:55:54.774506 ip-10-0-133-55 kubenswrapper[2563]: W0709 14:55:54.774466 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b633244_0920_4211_9f0a_66aeed3b2e2a.slice/crio-bdc0fea1b3f8de5bdf8796e9e8e9adae1ea41f54a0b451a1bdcd0dd93dc226f8 WatchSource:0}: Error finding container bdc0fea1b3f8de5bdf8796e9e8e9adae1ea41f54a0b451a1bdcd0dd93dc226f8: Status 404 returned error can't find the container with id bdc0fea1b3f8de5bdf8796e9e8e9adae1ea41f54a0b451a1bdcd0dd93dc226f8 Jul 09 14:55:55.509080 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:55.509033 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" event={"ID":"9b633244-0920-4211-9f0a-66aeed3b2e2a","Type":"ContainerStarted","Data":"2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd"} Jul 09 14:55:55.509501 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:55.509092 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" event={"ID":"9b633244-0920-4211-9f0a-66aeed3b2e2a","Type":"ContainerStarted","Data":"bdc0fea1b3f8de5bdf8796e9e8e9adae1ea41f54a0b451a1bdcd0dd93dc226f8"} Jul 09 14:55:55.509501 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:55.509113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:55:55.532261 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:55:55.532209 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" podStartSLOduration=2.192141205 podStartE2EDuration="2.532189696s" totalImagesPullingTime="340.048491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 14:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 14:55:55.532029581 +0000 UTC m=+2119.553112869" watchObservedRunningTime="2026-07-09 14:55:55.532189696 +0000 UTC m=+2119.553272962" Jul 09 14:56:00.479813 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:00.479772 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 14:56:00.480277 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:00.479838 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 14:56:01.520065 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:01.520037 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 14:56:01.544331 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:01.544147 2563 generic.go:356] "Generic (PLEG): container finished" podID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerID="2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac" exitCode=0 Jul 09 14:56:01.544331 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:01.544212 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" event={"ID":"abab67af-a29a-4ee5-985e-3a38c90c6ee2","Type":"ContainerDied","Data":"2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac"} Jul 09 14:56:01.544939 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:01.544912 2563 scope.go:117] "RemoveContainer" containerID="2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac" Jul 09 14:56:01.546152 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:01.546123 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:56:01.546219 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:01.546199 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:56:01.546275 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:01.546252 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:56:02.557366 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:02.557325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" event={"ID":"abab67af-a29a-4ee5-985e-3a38c90c6ee2","Type":"ContainerStarted","Data":"0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1"} Jul 09 14:56:03.519413 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:03.519379 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:56:03.519591 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:03.519525 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 09 14:56:03.519591 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:03.519571 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 09 14:56:04.520035 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:04.519992 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 09 14:56:04.520430 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:04.520049 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 09 14:56:05.519327 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:05.519290 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 09 14:56:05.519504 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:05.519373 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 09 14:56:06.522506 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:06.522469 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:56:06.523002 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:06.522826 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:56:06.523530 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:06.523508 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 14:56:08.499668 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.499612 2563 patch_prober.go:28] interesting pod/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 09 14:56:08.500102 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.499697 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 09 14:56:08.603555 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.603522 2563 generic.go:356] "Generic (PLEG): container finished" podID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerID="a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3" exitCode=0 Jul 09 14:56:08.603737 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.603575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" event={"ID":"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288","Type":"ContainerDied","Data":"a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3"} Jul 09 14:56:08.603967 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.603949 2563 scope.go:117] "RemoveContainer" containerID="a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3" Jul 09 14:56:08.604966 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.604940 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:56:08.605029 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.605002 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:56:08.605076 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:08.605042 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 14:56:09.611263 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:09.611229 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" event={"ID":"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288","Type":"ContainerStarted","Data":"83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83"} Jul 09 14:56:09.927133 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:09.927043 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:56:09.927308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:09.927238 2563 patch_prober.go:28] interesting pod/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 09 14:56:09.927308 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:09.927289 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 09 14:56:10.927991 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:10.927951 2563 patch_prober.go:28] interesting pod/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 09 14:56:10.928461 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:10.928012 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 09 14:56:11.926909 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:11.926880 2563 patch_prober.go:28] interesting pod/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 09 14:56:11.927099 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:11.926935 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 09 14:56:12.927425 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:12.927386 2563 patch_prober.go:28] interesting pod/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 09 14:56:12.927819 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:12.927440 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 09 14:56:13.930966 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:13.930938 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:56:13.931346 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:13.931174 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 14:56:13.931725 ip-10-0-133-55 kubenswrapper[2563]: I0709 14:56:13.931710 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 15:00:36.668312 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:00:36.668288 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 15:00:36.672773 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:00:36.672748 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 15:00:36.674215 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:00:36.674196 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 15:00:36.678881 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:00:36.678866 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 15:02:03.582021 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:03.581979 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n"] Jul 09 15:02:03.582652 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:03.582209 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" podUID="226842e0-dfde-4359-b361-77c62f77a5bd" containerName="maas-api" containerID="cri-o://04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074" gracePeriod=30 Jul 09 15:02:03.834154 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:03.834102 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 15:02:04.001647 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.001608 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls\") pod \"226842e0-dfde-4359-b361-77c62f77a5bd\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " Jul 09 15:02:04.001806 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.001698 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5hcd\" (UniqueName: \"kubernetes.io/projected/226842e0-dfde-4359-b361-77c62f77a5bd-kube-api-access-r5hcd\") pod \"226842e0-dfde-4359-b361-77c62f77a5bd\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " Jul 09 15:02:04.001806 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.001765 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226842e0-dfde-4359-b361-77c62f77a5bd-openshift-service-ca\") pod \"226842e0-dfde-4359-b361-77c62f77a5bd\" (UID: \"226842e0-dfde-4359-b361-77c62f77a5bd\") " Jul 09 15:02:04.002119 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.002091 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226842e0-dfde-4359-b361-77c62f77a5bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "226842e0-dfde-4359-b361-77c62f77a5bd" (UID: "226842e0-dfde-4359-b361-77c62f77a5bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:02:04.003714 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.003685 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "226842e0-dfde-4359-b361-77c62f77a5bd" (UID: "226842e0-dfde-4359-b361-77c62f77a5bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:02:04.003714 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.003687 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226842e0-dfde-4359-b361-77c62f77a5bd-kube-api-access-r5hcd" (OuterVolumeSpecName: "kube-api-access-r5hcd") pod "226842e0-dfde-4359-b361-77c62f77a5bd" (UID: "226842e0-dfde-4359-b361-77c62f77a5bd"). InnerVolumeSpecName "kube-api-access-r5hcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:02:04.103207 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.103153 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226842e0-dfde-4359-b361-77c62f77a5bd-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:04.103207 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.103175 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226842e0-dfde-4359-b361-77c62f77a5bd-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:04.103207 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.103183 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hcd\" (UniqueName: \"kubernetes.io/projected/226842e0-dfde-4359-b361-77c62f77a5bd-kube-api-access-r5hcd\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:04.560405 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.560370 2563 generic.go:356] "Generic (PLEG): container finished" podID="226842e0-dfde-4359-b361-77c62f77a5bd" containerID="04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074" exitCode=0 Jul 09 15:02:04.560580 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.560416 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" Jul 09 15:02:04.560580 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.560461 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" event={"ID":"226842e0-dfde-4359-b361-77c62f77a5bd","Type":"ContainerDied","Data":"04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074"} Jul 09 15:02:04.560580 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.560519 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n" event={"ID":"226842e0-dfde-4359-b361-77c62f77a5bd","Type":"ContainerDied","Data":"7c6df130937a58bb5ce667516040c77f996f471b18df9fc30ce4cbd247f2c029"} Jul 09 15:02:04.560580 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.560548 2563 scope.go:117] "RemoveContainer" containerID="04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074" Jul 09 15:02:04.569925 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.569907 2563 scope.go:117] "RemoveContainer" containerID="04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074" Jul 09 15:02:04.570185 ip-10-0-133-55 kubenswrapper[2563]: E0709 15:02:04.570166 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074\": container with ID starting with 04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074 not found: ID does not exist" containerID="04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074" Jul 09 15:02:04.570259 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.570197 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074"} err="failed to get container status \"04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074\": rpc error: code = NotFound desc = could not find container \"04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074\": container with ID starting with 04d5a8183859ce4a923d2b28ece712ca18f0469f425969b00705441a716d6074 not found: ID does not exist" Jul 09 15:02:04.583259 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.583238 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n"] Jul 09 15:02:04.589007 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:04.588988 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d877f-569797dc69-2957n"] Jul 09 15:02:06.575191 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:06.575163 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226842e0-dfde-4359-b361-77c62f77a5bd" path="/var/lib/kubelet/pods/226842e0-dfde-4359-b361-77c62f77a5bd/volumes" Jul 09 15:02:09.692871 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:09.692839 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t"] Jul 09 15:02:09.693315 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:09.693106 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" containerID="cri-o://0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1" gracePeriod=30 Jul 09 15:02:10.813637 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:10.813589 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br"] Jul 09 15:02:10.814106 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:10.813816 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" podUID="9b633244-0920-4211-9f0a-66aeed3b2e2a" containerName="maas-api" containerID="cri-o://2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd" gracePeriod=30 Jul 09 15:02:11.054466 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.054441 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 15:02:11.165724 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.165653 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls\") pod \"9b633244-0920-4211-9f0a-66aeed3b2e2a\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " Jul 09 15:02:11.165724 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.165690 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b633244-0920-4211-9f0a-66aeed3b2e2a-openshift-service-ca\") pod \"9b633244-0920-4211-9f0a-66aeed3b2e2a\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " Jul 09 15:02:11.165931 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.165735 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58pqq\" (UniqueName: \"kubernetes.io/projected/9b633244-0920-4211-9f0a-66aeed3b2e2a-kube-api-access-58pqq\") pod \"9b633244-0920-4211-9f0a-66aeed3b2e2a\" (UID: \"9b633244-0920-4211-9f0a-66aeed3b2e2a\") " Jul 09 15:02:11.166081 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.166053 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b633244-0920-4211-9f0a-66aeed3b2e2a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b633244-0920-4211-9f0a-66aeed3b2e2a" (UID: "9b633244-0920-4211-9f0a-66aeed3b2e2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:02:11.167701 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.167679 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b633244-0920-4211-9f0a-66aeed3b2e2a" (UID: "9b633244-0920-4211-9f0a-66aeed3b2e2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 15:02:11.167800 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.167744 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b633244-0920-4211-9f0a-66aeed3b2e2a-kube-api-access-58pqq" (OuterVolumeSpecName: "kube-api-access-58pqq") pod "9b633244-0920-4211-9f0a-66aeed3b2e2a" (UID: "9b633244-0920-4211-9f0a-66aeed3b2e2a"). InnerVolumeSpecName "kube-api-access-58pqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:02:11.266431 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.266399 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b633244-0920-4211-9f0a-66aeed3b2e2a-maas-api-tls\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:11.266431 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.266427 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b633244-0920-4211-9f0a-66aeed3b2e2a-openshift-service-ca\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:11.266431 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.266436 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58pqq\" (UniqueName: \"kubernetes.io/projected/9b633244-0920-4211-9f0a-66aeed3b2e2a-kube-api-access-58pqq\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:11.600124 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.600094 2563 generic.go:356] "Generic (PLEG): container finished" podID="9b633244-0920-4211-9f0a-66aeed3b2e2a" containerID="2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd" exitCode=0 Jul 09 15:02:11.600316 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.600136 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" Jul 09 15:02:11.600316 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.600169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" event={"ID":"9b633244-0920-4211-9f0a-66aeed3b2e2a","Type":"ContainerDied","Data":"2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd"} Jul 09 15:02:11.600316 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.600213 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br" event={"ID":"9b633244-0920-4211-9f0a-66aeed3b2e2a","Type":"ContainerDied","Data":"bdc0fea1b3f8de5bdf8796e9e8e9adae1ea41f54a0b451a1bdcd0dd93dc226f8"} Jul 09 15:02:11.600316 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.600234 2563 scope.go:117] "RemoveContainer" containerID="2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd" Jul 09 15:02:11.609418 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.609401 2563 scope.go:117] "RemoveContainer" containerID="2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd" Jul 09 15:02:11.609683 ip-10-0-133-55 kubenswrapper[2563]: E0709 15:02:11.609666 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd\": container with ID starting with 2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd not found: ID does not exist" containerID="2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd" Jul 09 15:02:11.609736 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.609692 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd"} err="failed to get container status \"2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd\": rpc error: code = NotFound desc = could not find container \"2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd\": container with ID starting with 2dbf1b40bdd35e44ec7290bca6de9ef11baa41ba8964d9d98e07992a4cabefdd not found: ID does not exist" Jul 09 15:02:11.622848 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.622824 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br"] Jul 09 15:02:11.628171 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:11.628151 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d74e21-5fcd869db5-595br"] Jul 09 15:02:12.574688 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:12.574658 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b633244-0920-4211-9f0a-66aeed3b2e2a" path="/var/lib/kubelet/pods/9b633244-0920-4211-9f0a-66aeed3b2e2a/volumes" Jul 09 15:02:14.946743 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.946716 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 15:02:14.998111 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998049 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-envoy\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998111 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998081 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-podinfo\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998303 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998137 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-certs\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998303 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998157 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istiod-ca-cert\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998303 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998179 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-token\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998303 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998196 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-credential-socket\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998303 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998227 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-socket\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998555 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998327 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-data\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998555 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998379 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qjw9\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-kube-api-access-7qjw9\") pod \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\" (UID: \"abab67af-a29a-4ee5-985e-3a38c90c6ee2\") " Jul 09 15:02:14.998555 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998381 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:14.998744 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998577 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:14.998744 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998600 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:02:14.998848 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998755 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:14.998848 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998773 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:14.998848 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998785 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:14.999004 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.998960 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:14.999199 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:14.999171 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-data" (OuterVolumeSpecName: "istio-data") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:15.000286 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.000262 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:15.000565 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.000542 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-kube-api-access-7qjw9" (OuterVolumeSpecName: "kube-api-access-7qjw9") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "kube-api-access-7qjw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:02:15.000565 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.000553 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 15:02:15.000667 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.000547 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-token" (OuterVolumeSpecName: "istio-token") pod "abab67af-a29a-4ee5-985e-3a38c90c6ee2" (UID: "abab67af-a29a-4ee5-985e-3a38c90c6ee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:02:15.099261 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.099229 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:15.099261 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.099257 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:15.099460 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.099270 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:15.099460 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.099282 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qjw9\" (UniqueName: \"kubernetes.io/projected/abab67af-a29a-4ee5-985e-3a38c90c6ee2-kube-api-access-7qjw9\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:15.099460 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.099293 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:15.099460 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.099304 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abab67af-a29a-4ee5-985e-3a38c90c6ee2-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:15.622919 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.622888 2563 generic.go:356] "Generic (PLEG): container finished" podID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerID="0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1" exitCode=0 Jul 09 15:02:15.623114 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.622938 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" Jul 09 15:02:15.623114 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.622946 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" event={"ID":"abab67af-a29a-4ee5-985e-3a38c90c6ee2","Type":"ContainerDied","Data":"0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1"} Jul 09 15:02:15.623114 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.622985 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t" event={"ID":"abab67af-a29a-4ee5-985e-3a38c90c6ee2","Type":"ContainerDied","Data":"32a00186ac9afbc10eb012a723434987260c0075a45bf17ebe1cb76dd667fca5"} Jul 09 15:02:15.623114 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.623005 2563 scope.go:117] "RemoveContainer" containerID="0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1" Jul 09 15:02:15.632496 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.632483 2563 scope.go:117] "RemoveContainer" containerID="2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac" Jul 09 15:02:15.640037 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.640019 2563 scope.go:117] "RemoveContainer" containerID="0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1" Jul 09 15:02:15.640256 ip-10-0-133-55 kubenswrapper[2563]: E0709 15:02:15.640238 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1\": container with ID starting with 0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1 not found: ID does not exist" containerID="0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1" Jul 09 15:02:15.640326 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.640265 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1"} err="failed to get container status \"0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1\": rpc error: code = NotFound desc = could not find container \"0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1\": container with ID starting with 0d93837df5dd1e2d8f51ea1a303207aa50eb705565cb02481176e8fafceebca1 not found: ID does not exist" Jul 09 15:02:15.640326 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.640280 2563 scope.go:117] "RemoveContainer" containerID="2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac" Jul 09 15:02:15.640487 ip-10-0-133-55 kubenswrapper[2563]: E0709 15:02:15.640472 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac\": container with ID starting with 2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac not found: ID does not exist" containerID="2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac" Jul 09 15:02:15.640529 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.640490 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac"} err="failed to get container status \"2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac\": rpc error: code = NotFound desc = could not find container \"2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac\": container with ID starting with 2f5a17b9a5b9ecc6c51664d2bbd4fcb6498dcf8d8142444802968e4338adc3ac not found: ID does not exist" Jul 09 15:02:15.644655 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.644631 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t"] Jul 09 15:02:15.648836 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:15.648816 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-0d877f-openshift-default-76f86cb84c-2xg2t"] Jul 09 15:02:16.574500 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:16.574459 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" path="/var/lib/kubelet/pods/abab67af-a29a-4ee5-985e-3a38c90c6ee2/volumes" Jul 09 15:02:16.887574 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:16.887495 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g"] Jul 09 15:02:16.887739 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:16.887702 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" containerID="cri-o://83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83" gracePeriod=30 Jul 09 15:02:22.125866 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.125840 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 15:02:22.149153 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149126 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-certs\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149382 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149357 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-socket\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149504 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149380 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:22.149504 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149410 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-envoy\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149504 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149439 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-token\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149504 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149466 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-credential-socket\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149748 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149546 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istiod-ca-cert\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149748 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149580 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-data\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149748 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149605 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-podinfo\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149748 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149660 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddgrd\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-kube-api-access-ddgrd\") pod \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\" (UID: \"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288\") " Jul 09 15:02:22.149748 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149741 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:22.150408 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149942 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-certs\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.150408 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.149958 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-workload-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.150408 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.150151 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 15:02:22.150408 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.150316 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:22.150765 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.150539 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-data" (OuterVolumeSpecName: "istio-data") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:22.152442 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.152416 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-kube-api-access-ddgrd" (OuterVolumeSpecName: "kube-api-access-ddgrd") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "kube-api-access-ddgrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:02:22.152551 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.152528 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 15:02:22.152640 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.152544 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 15:02:22.153904 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.153882 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-token" (OuterVolumeSpecName: "istio-token") pod "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" (UID: "87c0bfb8-e1f7-4f2d-bbe3-b298b4272288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 15:02:22.250810 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.250779 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddgrd\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-kube-api-access-ddgrd\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.250810 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.250805 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-envoy\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.250998 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.250818 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-token\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.250998 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.250829 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-credential-socket\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.250998 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.250840 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istiod-ca-cert\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.250998 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.250850 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-data\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.250998 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.250862 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288-istio-podinfo\") on node \"ip-10-0-133-55.ec2.internal\" DevicePath \"\"" Jul 09 15:02:22.660103 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.660074 2563 generic.go:356] "Generic (PLEG): container finished" podID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerID="83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83" exitCode=0 Jul 09 15:02:22.660213 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.660117 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" Jul 09 15:02:22.660213 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.660153 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" event={"ID":"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288","Type":"ContainerDied","Data":"83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83"} Jul 09 15:02:22.660213 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.660198 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g" event={"ID":"87c0bfb8-e1f7-4f2d-bbe3-b298b4272288","Type":"ContainerDied","Data":"1a701d5560a0368de918812e7af69790aa19d8215bb8b112c9e7a30be239154d"} Jul 09 15:02:22.660329 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.660225 2563 scope.go:117] "RemoveContainer" containerID="83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83" Jul 09 15:02:22.669026 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.669011 2563 scope.go:117] "RemoveContainer" containerID="a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3" Jul 09 15:02:22.678147 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.678121 2563 scope.go:117] "RemoveContainer" containerID="83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83" Jul 09 15:02:22.678468 ip-10-0-133-55 kubenswrapper[2563]: E0709 15:02:22.678445 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83\": container with ID starting with 83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83 not found: ID does not exist" containerID="83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83" Jul 09 15:02:22.678539 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.678476 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83"} err="failed to get container status \"83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83\": rpc error: code = NotFound desc = could not find container \"83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83\": container with ID starting with 83cb349ec39ceaf7aa80b0f81a1469568e52da29b8a4dedd010fcfdd7b48ab83 not found: ID does not exist" Jul 09 15:02:22.678539 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.678493 2563 scope.go:117] "RemoveContainer" containerID="a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3" Jul 09 15:02:22.678757 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.678737 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g"] Jul 09 15:02:22.678881 ip-10-0-133-55 kubenswrapper[2563]: E0709 15:02:22.678781 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3\": container with ID starting with a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3 not found: ID does not exist" containerID="a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3" Jul 09 15:02:22.678881 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.678801 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3"} err="failed to get container status \"a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3\": rpc error: code = NotFound desc = could not find container \"a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3\": container with ID starting with a871bd3d7e769608af7b01dd519ef6e02a1f12ca24a410fdca5a6f59574abaa3 not found: ID does not exist" Jul 09 15:02:22.684769 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.684749 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d74e21-openshift-default-565d76b6c9-ccd2g"] Jul 09 15:02:22.818744 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:22.818684 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dff748cfb-tfmfr_ada34cf0-f30d-4630-a359-64fb51d51f4d/manager/0.log" Jul 09 15:02:23.316539 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:23.316509 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7757d956-ngt74_396125ee-ee41-419c-87ff-a5e807f6079b/copy-manifests/0.log" Jul 09 15:02:23.323213 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:23.323193 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7757d956-ngt74_396125ee-ee41-419c-87ff-a5e807f6079b/manager/0.log" Jul 09 15:02:23.802376 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:23.802333 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57b5dcc795-n2c4h_7e479426-d010-4701-ad5a-1bda78affe3e/maas-api/0.log" Jul 09 15:02:24.581004 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.580972 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" path="/var/lib/kubelet/pods/87c0bfb8-e1f7-4f2d-bbe3-b298b4272288/volumes" Jul 09 15:02:24.716602 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.716579 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9_30db95bf-dec5-41c5-be6d-d8dfc9fda83f/util/0.log" Jul 09 15:02:24.723522 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.723501 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9_30db95bf-dec5-41c5-be6d-d8dfc9fda83f/pull/0.log" Jul 09 15:02:24.730339 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.730318 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9_30db95bf-dec5-41c5-be6d-d8dfc9fda83f/extract/0.log" Jul 09 15:02:24.842341 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.842283 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh_6a1edc03-ea74-40c5-b334-8489c611ca8e/util/0.log" Jul 09 15:02:24.848661 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.848641 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh_6a1edc03-ea74-40c5-b334-8489c611ca8e/pull/0.log" Jul 09 15:02:24.854830 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.854789 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh_6a1edc03-ea74-40c5-b334-8489c611ca8e/extract/0.log" Jul 09 15:02:24.969419 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.969394 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn_79a53746-c1b0-449f-bf4d-6551f622556f/util/0.log" Jul 09 15:02:24.977354 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.977337 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn_79a53746-c1b0-449f-bf4d-6551f622556f/pull/0.log" Jul 09 15:02:24.985971 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:24.985956 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn_79a53746-c1b0-449f-bf4d-6551f622556f/extract/0.log" Jul 09 15:02:25.100333 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:25.100260 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55_cb414b74-3776-4cfe-81d6-5fa5b37b6444/pull/0.log" Jul 09 15:02:25.106680 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:25.106663 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55_cb414b74-3776-4cfe-81d6-5fa5b37b6444/extract/0.log" Jul 09 15:02:25.113051 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:25.113035 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55_cb414b74-3776-4cfe-81d6-5fa5b37b6444/util/0.log" Jul 09 15:02:26.436892 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:26.436866 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42cjjx_41f155d6-09a0-4936-9a7d-8f1458ba9685/istio-proxy/0.log" Jul 09 15:02:26.558929 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:26.558876 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tvnvf_f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef/discovery/0.log" Jul 09 15:02:27.262478 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:27.262452 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bdf54bb-m5z9v_5a553e49-b8e4-4543-9550-9eba5165cc3a/router/0.log" Jul 09 15:02:27.601233 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:27.601144 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-25gbp_1f5b66c3-ee99-4958-a63e-26d3941de8fa/main/0.log" Jul 09 15:02:34.804998 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:34.804927 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-928hb_e527178e-badd-4625-a67a-f5e2904fbc82/global-pull-secret-syncer/0.log" Jul 09 15:02:34.925297 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:34.925261 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zmmhc_4991ea6f-5fcf-498a-8570-0817de45dd00/konnectivity-agent/0.log" Jul 09 15:02:34.946413 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:34.946388 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-55.ec2.internal_5340419ec45aff54cbe78dced08dd234/haproxy/0.log" Jul 09 15:02:39.737753 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.737723 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9_30db95bf-dec5-41c5-be6d-d8dfc9fda83f/extract/0.log" Jul 09 15:02:39.761978 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.761952 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9_30db95bf-dec5-41c5-be6d-d8dfc9fda83f/util/0.log" Jul 09 15:02:39.793454 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.793427 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759frsc9_30db95bf-dec5-41c5-be6d-d8dfc9fda83f/pull/0.log" Jul 09 15:02:39.823695 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.823670 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh_6a1edc03-ea74-40c5-b334-8489c611ca8e/extract/0.log" Jul 09 15:02:39.848080 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.848055 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh_6a1edc03-ea74-40c5-b334-8489c611ca8e/util/0.log" Jul 09 15:02:39.870664 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.870595 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpngh_6a1edc03-ea74-40c5-b334-8489c611ca8e/pull/0.log" Jul 09 15:02:39.910289 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.910258 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn_79a53746-c1b0-449f-bf4d-6551f622556f/extract/0.log" Jul 09 15:02:39.935137 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.935110 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn_79a53746-c1b0-449f-bf4d-6551f622556f/util/0.log" Jul 09 15:02:39.960965 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:39.960938 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jpmvn_79a53746-c1b0-449f-bf4d-6551f622556f/pull/0.log" Jul 09 15:02:40.005851 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:40.005772 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55_cb414b74-3776-4cfe-81d6-5fa5b37b6444/extract/0.log" Jul 09 15:02:40.029050 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:40.029027 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55_cb414b74-3776-4cfe-81d6-5fa5b37b6444/util/0.log" Jul 09 15:02:40.054355 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:40.054329 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rhl55_cb414b74-3776-4cfe-81d6-5fa5b37b6444/pull/0.log" Jul 09 15:02:42.735258 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:42.735223 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-4zmfm_1329b344-4e47-4d7a-b4c5-00fd4a60435e/cluster-monitoring-operator/0.log" Jul 09 15:02:43.166567 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:43.166540 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdv22_8a06c082-ca75-4ad7-aa94-6800b0c1d051/node-exporter/0.log" Jul 09 15:02:43.187966 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:43.187944 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdv22_8a06c082-ca75-4ad7-aa94-6800b0c1d051/kube-rbac-proxy/0.log" Jul 09 15:02:43.211844 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:43.211827 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdv22_8a06c082-ca75-4ad7-aa94-6800b0c1d051/init-textfile/0.log" Jul 09 15:02:45.015405 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:45.015355 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-wwjv6_25e5eacf-4b28-4bdb-aa9e-214257dfe58a/networking-console-plugin/0.log" Jul 09 15:02:45.618202 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:45.618169 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/2.log" Jul 09 15:02:45.622604 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:45.622584 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-bbxxs_6966a58b-4631-44d0-8087-1bf9f97bdab3/console-operator/3.log" Jul 09 15:02:46.116333 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.116294 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cfd7dc998-pj6gh_fcaebcad-6cca-4fbe-a120-11975ae53423/console/0.log" Jul 09 15:02:46.153472 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.153446 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-ljw9j_8711347f-783c-4d25-92ed-b1ef6095c77a/download-server/0.log" Jul 09 15:02:46.759011 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.758986 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-gm4nr_7761ed1a-569d-497c-8f38-af325db1a0cf/volume-data-source-validator/0.log" Jul 09 15:02:46.865649 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.865605 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk"] Jul 09 15:02:46.865970 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.865956 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" Jul 09 15:02:46.866019 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.865972 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" Jul 09 15:02:46.866019 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.865987 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" Jul 09 15:02:46.866019 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.865993 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" Jul 09 15:02:46.866019 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866002 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226842e0-dfde-4359-b361-77c62f77a5bd" containerName="maas-api" Jul 09 15:02:46.866019 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866008 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="226842e0-dfde-4359-b361-77c62f77a5bd" containerName="maas-api" Jul 09 15:02:46.866019 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866017 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b633244-0920-4211-9f0a-66aeed3b2e2a" containerName="maas-api" Jul 09 15:02:46.866019 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866022 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b633244-0920-4211-9f0a-66aeed3b2e2a" containerName="maas-api" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866032 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866037 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866045 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866050 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866106 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866114 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="87c0bfb8-e1f7-4f2d-bbe3-b298b4272288" containerName="istio-proxy" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866123 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b633244-0920-4211-9f0a-66aeed3b2e2a" containerName="maas-api" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866133 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866142 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="226842e0-dfde-4359-b361-77c62f77a5bd" containerName="maas-api" Jul 09 15:02:46.866264 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.866152 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abab67af-a29a-4ee5-985e-3a38c90c6ee2" containerName="istio-proxy" Jul 09 15:02:46.869289 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.869270 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:46.871761 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.871719 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ltzpn\"/\"kube-root-ca.crt\"" Jul 09 15:02:46.871876 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.871719 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ltzpn\"/\"openshift-service-ca.crt\"" Jul 09 15:02:46.871876 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.871828 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ltzpn\"/\"default-dockercfg-pdr4l\"" Jul 09 15:02:46.879809 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.879784 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk"] Jul 09 15:02:46.952690 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.952655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-proc\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:46.952863 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.952697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-lib-modules\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:46.952863 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.952726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq28\" (UniqueName: \"kubernetes.io/projected/80c310b9-d1b5-4ee2-94cd-5083b3286824-kube-api-access-gwq28\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:46.952863 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.952762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-sys\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:46.952863 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:46.952782 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-podres\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054198 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-proc\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054198 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-lib-modules\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054384 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-proc\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054384 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054287 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq28\" (UniqueName: \"kubernetes.io/projected/80c310b9-d1b5-4ee2-94cd-5083b3286824-kube-api-access-gwq28\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054384 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-sys\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054384 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054334 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-lib-modules\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054384 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054337 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-podres\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054546 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054400 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-sys\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.054546 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.054461 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80c310b9-d1b5-4ee2-94cd-5083b3286824-podres\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.062371 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.062346 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq28\" (UniqueName: \"kubernetes.io/projected/80c310b9-d1b5-4ee2-94cd-5083b3286824-kube-api-access-gwq28\") pod \"perf-node-gather-daemonset-p6pxk\" (UID: \"80c310b9-d1b5-4ee2-94cd-5083b3286824\") " pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.179837 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.179811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.305853 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.305829 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk"] Jul 09 15:02:47.307086 ip-10-0-133-55 kubenswrapper[2563]: W0709 15:02:47.307045 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80c310b9_d1b5_4ee2_94cd_5083b3286824.slice/crio-0313f8aa10141b5ef786b0439b74172dc1a90370f9ca9301e5522cd1e383340a WatchSource:0}: Error finding container 0313f8aa10141b5ef786b0439b74172dc1a90370f9ca9301e5522cd1e383340a: Status 404 returned error can't find the container with id 0313f8aa10141b5ef786b0439b74172dc1a90370f9ca9301e5522cd1e383340a Jul 09 15:02:47.786924 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.786879 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" event={"ID":"80c310b9-d1b5-4ee2-94cd-5083b3286824","Type":"ContainerStarted","Data":"e98a26ace4d8a14a3c8e02aaf2555fdfa2072d0e5e3af2fda57cfdfa943b59af"} Jul 09 15:02:47.787082 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.786966 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" event={"ID":"80c310b9-d1b5-4ee2-94cd-5083b3286824","Type":"ContainerStarted","Data":"0313f8aa10141b5ef786b0439b74172dc1a90370f9ca9301e5522cd1e383340a"} Jul 09 15:02:47.787082 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.786987 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:47.804130 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.804087 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" podStartSLOduration=1.804073731 podStartE2EDuration="1.804073731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 15:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 15:02:47.801835347 +0000 UTC m=+2531.822918612" watchObservedRunningTime="2026-07-09 15:02:47.804073731 +0000 UTC m=+2531.825156995" Jul 09 15:02:47.927024 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.926999 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fwcm_91ad122e-b32c-483b-89c3-fac94cbfc115/dns/0.log" Jul 09 15:02:47.950852 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:47.950828 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fwcm_91ad122e-b32c-483b-89c3-fac94cbfc115/kube-rbac-proxy/0.log" Jul 09 15:02:48.147693 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:48.147612 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sx47j_d6a5d9db-5636-48cd-bb96-76b36e182c4b/dns-node-resolver/0.log" Jul 09 15:02:48.739113 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:48.739086 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b4jwm_7ca81f21-58d3-478f-a38e-afd97d8aff04/node-ca/0.log" Jul 09 15:02:49.746828 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:49.746800 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42cjjx_41f155d6-09a0-4936-9a7d-8f1458ba9685/istio-proxy/0.log" Jul 09 15:02:49.858814 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:49.858781 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tvnvf_f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef/discovery/0.log" Jul 09 15:02:50.086890 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:50.086863 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bdf54bb-m5z9v_5a553e49-b8e4-4543-9550-9eba5165cc3a/router/0.log" Jul 09 15:02:50.792637 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:50.792592 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-27qjw_9889133b-8a20-4733-a2ed-ea4f39c13ece/serve-healthcheck-canary/0.log" Jul 09 15:02:51.346911 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:51.346880 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-fwnfg_7ae68662-8dc0-48d2-ad61-b6e631f5fc81/insights-operator/0.log" Jul 09 15:02:51.347441 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:51.347416 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-fwnfg_7ae68662-8dc0-48d2-ad61-b6e631f5fc81/insights-operator/1.log" Jul 09 15:02:51.446281 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:51.446256 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jqssn_51cf2858-a3fa-429c-b87c-ebb15481fdb1/kube-rbac-proxy/0.log" Jul 09 15:02:51.468767 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:51.468742 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jqssn_51cf2858-a3fa-429c-b87c-ebb15481fdb1/exporter/0.log" Jul 09 15:02:51.492610 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:51.492556 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jqssn_51cf2858-a3fa-429c-b87c-ebb15481fdb1/extractor/0.log" Jul 09 15:02:53.804471 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:53.804447 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ltzpn/perf-node-gather-daemonset-p6pxk" Jul 09 15:02:54.016006 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:54.015973 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dff748cfb-tfmfr_ada34cf0-f30d-4630-a359-64fb51d51f4d/manager/0.log" Jul 09 15:02:54.228566 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:54.228487 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7757d956-ngt74_396125ee-ee41-419c-87ff-a5e807f6079b/manager/0.log" Jul 09 15:02:54.251486 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:54.251458 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7757d956-ngt74_396125ee-ee41-419c-87ff-a5e807f6079b/copy-manifests/0.log" Jul 09 15:02:55.084170 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:55.084140 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42cjjx_41f155d6-09a0-4936-9a7d-8f1458ba9685/istio-proxy/0.log" Jul 09 15:02:55.180749 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:55.180715 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tvnvf_f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef/discovery/0.log" Jul 09 15:02:55.354714 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:55.354612 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bdf54bb-m5z9v_5a553e49-b8e4-4543-9550-9eba5165cc3a/router/0.log" Jul 09 15:02:55.658823 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:55.658742 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7988959d95-v4gs6_c7594106-a2f2-44ba-9d91-40d12f177f95/manager/0.log" Jul 09 15:02:55.691243 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:55.691214 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2k8zx_e761aa26-971b-44b8-95ee-503f96dc06b9/openshift-lws-operator/0.log" Jul 09 15:02:56.742952 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:56.742920 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42cjjx_41f155d6-09a0-4936-9a7d-8f1458ba9685/istio-proxy/0.log" Jul 09 15:02:56.848093 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:56.848060 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tvnvf_f1f09d0e-a4d0-4463-9bcb-c41430ebe4ef/discovery/0.log" Jul 09 15:02:57.031660 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:02:57.031631 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5bdf54bb-m5z9v_5a553e49-b8e4-4543-9550-9eba5165cc3a/router/0.log" Jul 09 15:03:00.836333 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:00.836301 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-55jxj_04cbc9f6-c3d3-4509-8449-cc0b191e212a/migrator/0.log" Jul 09 15:03:00.864750 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:00.864720 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-55jxj_04cbc9f6-c3d3-4509-8449-cc0b191e212a/graceful-termination/0.log" Jul 09 15:03:01.256111 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:01.256087 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-88h2b_1858e3a2-d403-45fc-88d3-d18aca4057c6/kube-storage-version-migrator-operator/1.log" Jul 09 15:03:01.256865 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:01.256844 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-88h2b_1858e3a2-d403-45fc-88d3-d18aca4057c6/kube-storage-version-migrator-operator/0.log" Jul 09 15:03:02.462462 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.462433 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnrwr_c5c393e5-2ad0-433a-9373-ff8aea28ed0c/kube-multus-additional-cni-plugins/0.log" Jul 09 15:03:02.488332 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.488312 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnrwr_c5c393e5-2ad0-433a-9373-ff8aea28ed0c/egress-router-binary-copy/0.log" Jul 09 15:03:02.515392 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.515371 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnrwr_c5c393e5-2ad0-433a-9373-ff8aea28ed0c/cni-plugins/0.log" Jul 09 15:03:02.540954 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.540937 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnrwr_c5c393e5-2ad0-433a-9373-ff8aea28ed0c/bond-cni-plugin/0.log" Jul 09 15:03:02.564953 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.564938 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnrwr_c5c393e5-2ad0-433a-9373-ff8aea28ed0c/routeoverride-cni/0.log" Jul 09 15:03:02.588215 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.588194 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnrwr_c5c393e5-2ad0-433a-9373-ff8aea28ed0c/whereabouts-cni-bincopy/0.log" Jul 09 15:03:02.613172 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.613156 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jnrwr_c5c393e5-2ad0-433a-9373-ff8aea28ed0c/whereabouts-cni/0.log" Jul 09 15:03:02.887887 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.887865 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k49pd_df7474eb-d84f-475a-a445-21a49a350284/kube-multus/0.log" Jul 09 15:03:02.961413 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.961395 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fb2rh_4154366d-8c15-403c-8df9-cceea3424319/network-metrics-daemon/0.log" Jul 09 15:03:02.984981 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:02.984965 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fb2rh_4154366d-8c15-403c-8df9-cceea3424319/kube-rbac-proxy/0.log" Jul 09 15:03:04.353025 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.353000 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-controller/0.log" Jul 09 15:03:04.375098 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.375077 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/1.log" Jul 09 15:03:04.375303 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.375287 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/ovn-acl-logging/0.log" Jul 09 15:03:04.395138 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.395117 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/kube-rbac-proxy-node/0.log" Jul 09 15:03:04.418613 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.418592 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/kube-rbac-proxy-ovn-metrics/0.log" Jul 09 15:03:04.441084 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.441064 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/northd/0.log" Jul 09 15:03:04.465170 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.465146 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/nbdb/0.log" Jul 09 15:03:04.493182 ip-10-0-133-55 kubenswrapper[2563]: I0709 15:03:04.493161 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rzbg4_b3aca0a7-2162-4481-bd9d-f2dee1a6fa33/sbdb/0.log"