Jul 06 19:21:24.833191 ip-10-0-131-106 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:21:24.833201 ip-10-0-131-106 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:21:24.833208 ip-10-0-131-106 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:21:24.833418 ip-10-0-131-106 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:21:35.065694 ip-10-0-131-106 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:21:35.065713 ip-10-0-131-106 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5c7c11b2a0004d0e8817deed0ccdb028 -- Jul 06 19:23:10.231936 ip-10-0-131-106 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.669107 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672934 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672945 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672948 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672951 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672955 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672958 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672961 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:23:10.801981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672971 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:23:10.738977 ip-10-0-131-106 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672974 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672978 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672980 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672985 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672989 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672992 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672995 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.672997 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673000 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673003 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673005 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673009 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673012 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673016 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673019 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673022 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673025 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673028 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:23:10.817142 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673030 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673033 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673036 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673038 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673041 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673044 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673047 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673050 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673053 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673055 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673058 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673060 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673063 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673072 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673074 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673077 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673081 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673084 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673087 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:23:10.968200 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673089 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673092 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673094 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673097 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673099 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673103 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673105 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673108 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673110 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673113 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673116 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673118 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673121 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673123 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673125 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673128 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673131 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673133 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673136 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673138 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:23:10.970828 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673141 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673143 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673146 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673148 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673151 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673153 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673156 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673165 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673168 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673170 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673173 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673175 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673178 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673180 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673183 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673185 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673188 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673190 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673192 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673198 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:23:10.972794 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673201 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673203 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673206 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673208 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673211 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673213 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673216 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673218 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673221 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673223 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673226 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673228 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673231 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.673234 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675937 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675945 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675948 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675952 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675956 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675959 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:23:10.975159 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675962 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675965 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675968 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675971 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675975 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675978 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675982 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675987 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675991 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675995 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.675998 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676000 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676003 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676005 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676008 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676010 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676013 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676016 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676019 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:23:10.977066 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676022 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676025 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676027 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676030 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676032 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676036 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676038 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676041 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676043 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676046 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676048 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676051 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676054 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676056 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676059 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676062 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676065 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676068 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676070 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676075 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:23:10.977792 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676078 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676082 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676085 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676088 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676092 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676095 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676098 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676101 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676104 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676106 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676109 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676111 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676114 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676116 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676119 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676122 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676124 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676126 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676129 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:23:10.978409 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676131 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676133 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676136 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676138 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676141 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676144 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676146 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676148 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676151 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676156 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676159 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676161 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676164 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676166 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676169 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676172 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676174 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676177 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676179 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676183 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:23:10.979122 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676186 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676188 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676191 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676195 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676199 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676202 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676204 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676207 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676209 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676212 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676214 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676217 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676220 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676223 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676226 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676580 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676585 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676587 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676590 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676593 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:23:10.979799 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676599 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676603 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676608 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676612 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676616 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676619 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676621 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676624 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676626 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676629 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676632 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676634 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676637 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676640 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676642 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676645 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676647 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676665 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676669 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:23:10.981807 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676673 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676679 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676683 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676686 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676688 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676692 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676694 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676697 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676700 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676705 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676707 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676710 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676713 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676716 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676720 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676722 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676725 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676728 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676730 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:23:10.983162 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676732 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676735 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676737 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676740 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676743 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676745 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676748 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676751 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676753 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676755 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676759 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676762 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676765 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676767 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676770 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676772 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676775 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676777 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676779 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676782 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:23:10.985513 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676784 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676786 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676790 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676793 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676795 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676798 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676800 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676804 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676807 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676809 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676812 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676816 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676820 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676824 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676828 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676832 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676835 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676837 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676840 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676842 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:23:10.987490 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676846 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676849 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676852 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676855 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676858 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676861 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676863 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676866 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676869 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676872 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676874 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676877 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676879 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676883 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676888 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.676904 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676964 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676970 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676976 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676981 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676985 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:23:10.989629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676989 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676993 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.676997 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677000 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677004 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677007 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677010 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677013 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677016 2585 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677019 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677024 2585 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677028 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677032 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677037 2585 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677040 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677057 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677062 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677066 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677070 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677074 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677078 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677082 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677086 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677089 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:23:10.990433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677093 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677098 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677101 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677106 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677110 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677113 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677118 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677121 2585 flags.go:64] FLAG: --enable-server="true" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677125 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677131 2585 flags.go:64] FLAG: --event-burst="100" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677135 2585 flags.go:64] FLAG: --event-qps="50" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677138 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677142 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677145 2585 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677150 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677153 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677157 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677161 2585 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677166 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677170 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677174 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677178 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677183 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677187 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677191 2585 flags.go:64] FLAG: --feature-gates="" Jul 06 19:23:10.992637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677196 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677200 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677205 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677209 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677214 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677218 2585 flags.go:64] FLAG: --help="false" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677223 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-131-106.ec2.internal" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677227 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677233 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677238 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677243 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677248 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677252 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677258 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677262 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677266 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677271 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677277 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677282 2585 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677287 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677292 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677296 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677301 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677306 2585 flags.go:64] FLAG: --lock-file="" Jul 06 19:23:10.994321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677310 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677315 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677320 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677330 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677334 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677339 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677343 2585 flags.go:64] FLAG: --logging-format="text" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677349 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677354 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677358 2585 flags.go:64] FLAG: --manifest-url="" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677362 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677368 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677373 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677379 2585 flags.go:64] FLAG: --max-pods="110" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677383 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677388 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677393 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677396 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677399 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677402 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677405 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677413 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677416 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677419 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:23:10.995222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677422 2585 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677424 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677431 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677433 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677436 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677439 2585 flags.go:64] FLAG: --port="10250" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677442 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677445 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bccdb22afbe33dff" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677448 2585 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677451 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677454 2585 flags.go:64] FLAG: --register-node="true" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677456 2585 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677460 2585 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677462 2585 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677465 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677468 2585 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677472 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677475 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677477 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677480 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677483 2585 flags.go:64] FLAG: --runonce="false" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677486 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677488 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677491 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677494 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:23:10.998316 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677498 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677502 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677504 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677507 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677511 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677514 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677516 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677519 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677522 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677525 2585 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677528 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677533 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677536 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677538 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677542 2585 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677545 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677548 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677550 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677553 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677556 2585 flags.go:64] FLAG: --v="2" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677560 2585 flags.go:64] FLAG: --version="false" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677564 2585 flags.go:64] FLAG: --vmodule="" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677567 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.677570 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677702 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:23:10.999121 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677707 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677710 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677713 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677716 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677721 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677724 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677727 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677731 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677734 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677737 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677740 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677743 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677746 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677748 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677751 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677753 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677756 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677759 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677762 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677764 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:23:11.000686 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677767 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677769 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677772 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677774 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677777 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677779 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677781 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677784 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677786 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677789 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677791 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677794 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677797 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677799 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677802 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677804 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677808 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677813 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677816 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677819 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:23:11.001252 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677822 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677826 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677829 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677834 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677837 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677839 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677842 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677845 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677847 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677850 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677853 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677856 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677859 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677861 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677864 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677867 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677870 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677874 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677877 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:23:11.188245 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677879 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677882 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677884 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677886 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677889 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677891 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677894 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677896 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677899 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677901 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677904 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677906 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677910 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677913 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677915 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677918 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677922 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677924 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677927 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677929 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:23:11.188928 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677932 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677934 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677937 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677939 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677942 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677945 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677947 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677950 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677952 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677955 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677957 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677959 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677962 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677965 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677967 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677970 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677972 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677975 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:23:11.189464 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.677978 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.678929 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.686286 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.686300 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.686325 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.686332 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686399 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686404 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686408 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686412 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686415 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686419 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686423 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686426 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:23:11.190005 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686429 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686431 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686434 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686437 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686439 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686442 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686444 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686447 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686450 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686453 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686456 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686459 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686461 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686465 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686468 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686470 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686473 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686475 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686478 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686480 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:23:11.190395 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686483 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686485 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686488 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686491 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686493 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686496 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686499 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686501 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686503 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686506 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686511 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686513 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686516 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686519 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686521 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686524 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686527 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686531 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686534 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:23:11.190981 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686537 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686540 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686543 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686547 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686549 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686552 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686555 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686557 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686560 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686563 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686565 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686568 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686570 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686573 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686576 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686578 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686581 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686583 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686586 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686588 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:23:11.191476 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686591 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686593 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686595 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686599 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686602 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686604 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686607 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686609 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686612 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686615 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686617 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686620 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686623 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686626 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686628 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686631 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686634 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686637 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686639 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:23:11.192109 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686642 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686644 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686647 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686649 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686673 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686676 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686678 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686681 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686684 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686686 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686689 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686691 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686694 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.686701 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686840 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:23:11.192606 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686845 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686849 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686854 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686857 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686860 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686863 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686866 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686870 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686873 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686876 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686879 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686883 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686886 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686888 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686891 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686894 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686896 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686903 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686906 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:23:11.193071 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686909 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686912 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686914 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686917 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686919 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686922 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686924 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686927 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686930 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686932 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686935 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686941 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686944 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686946 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686949 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686953 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686955 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686958 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686960 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686963 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:23:11.193574 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686966 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686968 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686971 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686973 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686976 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686979 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686981 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686984 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686987 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686989 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686992 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686995 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.686998 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687000 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687003 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687005 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687008 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687010 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687013 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687015 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:23:11.194151 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687018 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687021 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687023 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687027 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687029 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687032 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687034 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687038 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687041 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687043 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687046 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687048 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687051 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687053 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687056 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687058 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687061 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687063 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687067 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687071 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:23:11.194714 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687074 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687077 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687081 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687084 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687086 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687089 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687091 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687094 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687096 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687099 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687101 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687104 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687106 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687108 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687111 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687115 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687118 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687120 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:23:11.195286 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:10.687123 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.687129 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.687239 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.689321 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.690769 2585 server.go:1019] "Starting client certificate rotation" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.690869 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.690906 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.693787 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.695110 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.704720 2585 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.707288 2585 log.go:25] "Validated CRI v1 image API" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.708943 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:23:11.195826 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.711481 2585 fs.go:135] Filesystem UUIDs: map[3a0c2599-1194-4482-ba89-a8a9f24e9a0d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fc368d24-3473-486c-a83f-f2db28f76fd7:/dev/nvme0n1p4] Jul 06 19:23:11.196209 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.711495 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:23:11.196279 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.718581 2585 manager.go:216] Machine: {Timestamp:2026-07-06 19:23:10.717198806 +0000 UTC m=+0.359433623 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098938 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2869eb0d2552b3991df1662fc41cd6 SystemUUID:ec2869eb-0d25-52b3-991d-f1662fc41cd6 BootID:5c7c11b2-a000-4d0e-8817-deed0ccdb028 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:8a:f9:7f:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:8a:f9:7f:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:90:8e:50:18:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:23:11.196279 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.718694 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:23:11.196279 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.718766 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:23:11.196279 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.722141 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:23:11.196279 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.722168 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-106.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:23:11.196279 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.722617 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:23:11.196279 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.722910 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.722922 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.722930 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.723519 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.724901 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.727990 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728102 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728129 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728138 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728151 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728160 2585 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728169 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728464 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.728484 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.730669 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.732913 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.732939 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734116 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734129 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734135 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734140 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734148 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734154 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734164 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734169 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734175 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734181 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734188 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.734193 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.735167 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.735173 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.738130 2585 server.go:1268] "Started kubelet" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.738225 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.738292 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.738918 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.739322 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.739427 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-106.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.739443 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.739456 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:23:11.196554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.743900 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.744315 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.745808 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.745826 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.745906 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746388 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746405 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746390 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746530 2585 factory.go:55] Registering systemd factory Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746545 2585 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746554 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746565 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746788 2585 factory.go:153] Registering CRI-O factory Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746802 2585 factory.go:223] Registration of the crio container factory successfully Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746848 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746871 2585 factory.go:103] Registering Raw factory Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.746886 2585 manager.go:1196] Started watching for new ooms in manager Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.747218 2585 manager.go:349] Starting recovery of all containers Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.746763 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-106.ec2.internal.18bfc9bced873067 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-106.ec2.internal,UID:ip-10-0-131-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-106.ec2.internal,},FirstTimestamp:2026-07-06 19:23:10.738100327 +0000 UTC m=+0.380335148,LastTimestamp:2026-07-06 19:23:10.738100327 +0000 UTC m=+0.380335148,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-106.ec2.internal,}" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.753970 2585 manager.go:354] Recovery completed Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.755618 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.755765 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.758321 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.762190 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.762213 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.762223 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.762720 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.762732 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.762744 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.762749 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.763561 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.763581 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.763590 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.197853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.764026 2585 policy_none.go:49] "None policy: Start" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.764039 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.764049 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.765039 2585 policy_none.go:47] "Start" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.772332 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-106.ec2.internal.18bfc9bceef6f14f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-106.ec2.internal,UID:ip-10-0-131-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-106.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-106.ec2.internal,},FirstTimestamp:2026-07-06 19:23:10.762201423 +0000 UTC m=+0.404436240,LastTimestamp:2026-07-06 19:23:10.762201423 +0000 UTC m=+0.404436240,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-106.ec2.internal,}" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.778980 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlwrk" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.779329 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-106.ec2.internal.18bfc9bceef7317d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-106.ec2.internal,UID:ip-10-0-131-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-106.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-106.ec2.internal,},FirstTimestamp:2026-07-06 19:23:10.762217853 +0000 UTC m=+0.404452669,LastTimestamp:2026-07-06 19:23:10.762217853 +0000 UTC m=+0.404452669,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-106.ec2.internal,}" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.788114 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlwrk" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.789959 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-106.ec2.internal.18bfc9bceef75498 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-106.ec2.internal,UID:ip-10-0-131-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-106.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-106.ec2.internal,},FirstTimestamp:2026-07-06 19:23:10.76222684 +0000 UTC m=+0.404461656,LastTimestamp:2026-07-06 19:23:10.76222684 +0000 UTC m=+0.404461656,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-106.ec2.internal,}" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.843320 2585 manager.go:337] "Starting Device Plugin manager" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.843353 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.843363 2585 server.go:87] "Starting device plugin registration server" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.843648 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.843677 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.843760 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.843839 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.843849 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.844291 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.844333 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.864812 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:23:11.198797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.866140 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.866184 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.866208 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.866245 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.869208 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.944713 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.945603 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.945631 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.945644 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.945696 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.959780 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:10.959796 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-106.ec2.internal\": node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.966554 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal"] Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.966617 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.967979 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.968003 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.968013 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.969107 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.969237 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.969276 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.969902 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.969923 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.969935 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.970155 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.970170 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.970181 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.971334 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.971357 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.972873 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.972900 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:10.972915 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.008230 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-106.ec2.internal\" not found" node="ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.015686 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-106.ec2.internal\" not found" node="ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.345228 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.033303 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.133571 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.147964 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee390bc22bbe9a006e589c9e7a83c0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal\" (UID: \"dee390bc22bbe9a006e589c9e7a83c0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.147986 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/188ef7b029f22b3628e6f2183332d139-config\") pod \"kube-apiserver-proxy-ip-10-0-131-106.ec2.internal\" (UID: \"188ef7b029f22b3628e6f2183332d139\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.148008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dee390bc22bbe9a006e589c9e7a83c0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal\" (UID: \"dee390bc22bbe9a006e589c9e7a83c0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.233904 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.248243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dee390bc22bbe9a006e589c9e7a83c0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal\" (UID: \"dee390bc22bbe9a006e589c9e7a83c0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.248264 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee390bc22bbe9a006e589c9e7a83c0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal\" (UID: \"dee390bc22bbe9a006e589c9e7a83c0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.248281 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/188ef7b029f22b3628e6f2183332d139-config\") pod \"kube-apiserver-proxy-ip-10-0-131-106.ec2.internal\" (UID: \"188ef7b029f22b3628e6f2183332d139\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.248326 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/188ef7b029f22b3628e6f2183332d139-config\") pod \"kube-apiserver-proxy-ip-10-0-131-106.ec2.internal\" (UID: \"188ef7b029f22b3628e6f2183332d139\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.248333 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dee390bc22bbe9a006e589c9e7a83c0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal\" (UID: \"dee390bc22bbe9a006e589c9e7a83c0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.248357 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee390bc22bbe9a006e589c9e7a83c0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal\" (UID: \"dee390bc22bbe9a006e589c9e7a83c0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.311384 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.319179 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" Jul 06 19:23:11.346874 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.334611 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.518497 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.435108 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.699391 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.535638 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.699391 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.636102 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.699391 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.690606 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:23:11.699391 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.690736 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.737037 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.792840 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:18:11 +0000 UTC" deadline="2027-12-06 04:08:08.955896838 +0000 UTC" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.792903 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12416h44m57.163032351s" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.802012 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.803441 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.803483 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.803502 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.832650 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:23:11.850968 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.837773 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:12.019330 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:11.908496 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188ef7b029f22b3628e6f2183332d139.slice/crio-ebff42c66d5d8f32c82f5101636496f801aa1200132d12cc7539722376131554 WatchSource:0}: Error finding container ebff42c66d5d8f32c82f5101636496f801aa1200132d12cc7539722376131554: Status 404 returned error can't find the container with id ebff42c66d5d8f32c82f5101636496f801aa1200132d12cc7539722376131554 Jul 06 19:23:12.019330 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:11.908744 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee390bc22bbe9a006e589c9e7a83c0e.slice/crio-f079a233df1bd28bde88bd2059c1dce610008c8171864493dd98b8aca6bd490f WatchSource:0}: Error finding container f079a233df1bd28bde88bd2059c1dce610008c8171864493dd98b8aca6bd490f: Status 404 returned error can't find the container with id f079a233df1bd28bde88bd2059c1dce610008c8171864493dd98b8aca6bd490f Jul 06 19:23:12.019330 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:11.914654 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:23:12.019330 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:11.938594 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:12.224121 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.039208 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:12.224121 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.146426 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-106.ec2.internal\" not found" Jul 06 19:23:12.224121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.149189 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:23:12.358532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.246629 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" Jul 06 19:23:12.358532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.255940 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:23:12.358532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.280412 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:23:12.358532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.282372 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" Jul 06 19:23:12.358532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.293874 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.729974 2585 apiserver.go:52] "Watching apiserver" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.741419 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.742908 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal","openshift-multus/multus-additional-cni-plugins-cbh86","openshift-multus/multus-tbdks","openshift-network-operator/iptables-alerter-8s766","kube-system/global-pull-secret-syncer-6mcrp","openshift-cluster-node-tuning-operator/tuned-7jdkc","openshift-image-registry/node-ca-59ptz","openshift-multus/network-metrics-daemon-4m86h","openshift-network-diagnostics/network-check-target-scltl","openshift-ovn-kubernetes/ovnkube-node-g2flh","kube-system/konnectivity-agent-vkf9p","kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4"] Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.744783 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.747134 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.747236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.747685 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.748445 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.748485 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hd2z8\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.748487 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.749362 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.750579 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nrvwq\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.750643 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.750857 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.750892 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.751103 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.751133 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.751370 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.752316 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.752374 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5b8b9\"" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.752954 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.753013 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.754063 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.755419 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.755486 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:12.803808 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.757015 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.757075 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758470 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfldc\" (UniqueName: \"kubernetes.io/projected/d5c1554c-0f80-4c27-bc42-81bb12d90af9-kube-api-access-bfldc\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-os-release\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758525 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758552 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758575 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758577 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e265b652-468c-46c6-8d9e-8c48ae290882-host\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758813 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-system-cni-dir\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758840 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-cnibin\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-system-cni-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-netns\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758914 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnzw\" (UniqueName: \"kubernetes.io/projected/e265b652-468c-46c6-8d9e-8c48ae290882-kube-api-access-dmnzw\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-os-release\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.806570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.758990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-socket-dir-parent\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759028 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-cni-bin\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-cni-multus\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759102 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5c1554c-0f80-4c27-bc42-81bb12d90af9-cni-binary-copy\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-k8s-cni-cncf-io\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759149 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-conf-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759173 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-daemon-config\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-multus-certs\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759222 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvxq\" (UniqueName: \"kubernetes.io/projected/1b22434e-c80c-469d-9afd-0669319cbf91-kube-api-access-gxvxq\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759259 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-etc-kubernetes\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e265b652-468c-46c6-8d9e-8c48ae290882-serviceca\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759388 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759469 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-cni-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.807558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-cnibin\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759503 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-kubelet\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.759534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-hostroot\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.760146 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.761957 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762198 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762580 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762621 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762722 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762782 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762623 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762986 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.762996 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q8jwm\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.763018 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.763280 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.763379 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.763506 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.763597 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dh85b\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.763921 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.764051 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r6j4f\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.765106 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-krkft\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.765187 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.765297 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s87gk\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.765773 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.765891 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.765968 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.769380 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.789325 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fcb5n" Jul 06 19:23:12.810372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.805208 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fcb5n" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.830562 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.848172 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860444 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860472 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e265b652-468c-46c6-8d9e-8c48ae290882-host\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-cnibin\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860616 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-cnibin\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860623 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e265b652-468c-46c6-8d9e-8c48ae290882-host\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860656 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-socket-dir-parent\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860709 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-cni-bin\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860744 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-systemd-units\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860790 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860819 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-socket-dir-parent\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860833 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5c1554c-0f80-4c27-bc42-81bb12d90af9-cni-binary-copy\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.900689 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860855 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-cni-bin\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-k8s-cni-cncf-io\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860901 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-k8s-cni-cncf-io\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860898 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-daemon-config\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-multus-certs\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.860993 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-systemd\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861018 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-etc-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-kubelet-config\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861070 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-modprobe-d\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysconfig\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861142 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovnkube-script-lib\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861171 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvxq\" (UniqueName: \"kubernetes.io/projected/1b22434e-c80c-469d-9afd-0669319cbf91-kube-api-access-gxvxq\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-host-slash\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861219 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-tmp\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.904075 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861244 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-sys-fs\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e265b652-468c-46c6-8d9e-8c48ae290882-serviceca\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861300 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861387 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861394 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5c1554c-0f80-4c27-bc42-81bb12d90af9-cni-binary-copy\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861430 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-cni-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861455 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-cnibin\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861479 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-kubernetes\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861505 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-tuned\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861530 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-systemd\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861544 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861566 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861580 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-multus-certs\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861592 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-var-lib-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.904932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861621 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-log-socket\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861646 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovnkube-config\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861751 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-etc-selinux\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861827 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-kubelet\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psggn\" (UniqueName: \"kubernetes.io/projected/c5a5d8c4-8f2f-42eb-8113-0002828bee60-kube-api-access-psggn\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-registration-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861906 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-system-cni-dir\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861931 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-system-cni-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861956 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-daemon-config\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-netns\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.861994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-iptables-alerter-script\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862012 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-run-netns\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862014 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:12.906296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862046 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-run-netns\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862065 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-cni-bin\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862085 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnzw\" (UniqueName: \"kubernetes.io/projected/e265b652-468c-46c6-8d9e-8c48ae290882-kube-api-access-dmnzw\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-os-release\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862119 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-cni-multus\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862134 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-sys\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862150 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-env-overrides\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862190 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3880b-47bb-41bd-a6f6-d2eadcc62027-konnectivity-ca\") pod \"konnectivity-agent-vkf9p\" (UID: \"e7c3880b-47bb-41bd-a6f6-d2eadcc62027\") " pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862208 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-conf-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862230 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysctl-d\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcczl\" (UniqueName: \"kubernetes.io/projected/97ccad0f-176b-4351-8c9d-25cd80cedd9d-kube-api-access-dcczl\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovn-node-metrics-cert\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862289 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdgm\" (UniqueName: \"kubernetes.io/projected/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-kube-api-access-wmdgm\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862307 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-lib-modules\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.907147 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862414 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-ovn\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-etc-kubernetes\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862484 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysctl-conf\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862508 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-var-lib-kubelet\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862535 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-slash\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862577 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-kubelet\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862577 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863102 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-cni-binary-copy\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863163 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e265b652-468c-46c6-8d9e-8c48ae290882-serviceca\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863187 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-system-cni-dir\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863299 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-system-cni-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.862597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-hostroot\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863346 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfldc\" (UniqueName: \"kubernetes.io/projected/d5c1554c-0f80-4c27-bc42-81bb12d90af9-kube-api-access-bfldc\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.908348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863375 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-cni-netd\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863406 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248fk\" (UniqueName: \"kubernetes.io/projected/c7ce2adb-2188-42ad-9abb-348cb5444ce9-kube-api-access-248fk\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-os-release\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863468 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-run\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-host\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863520 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgkr\" (UniqueName: \"kubernetes.io/projected/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-kube-api-access-jrgkr\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863530 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b22434e-c80c-469d-9afd-0669319cbf91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863547 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863573 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7c3880b-47bb-41bd-a6f6-d2eadcc62027-agent-certs\") pod \"konnectivity-agent-vkf9p\" (UID: \"e7c3880b-47bb-41bd-a6f6-d2eadcc62027\") " pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-socket-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863625 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-dbus\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863649 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-cni-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863650 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-node-log\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863748 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-hostroot\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.909217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863897 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-etc-kubernetes\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863911 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-kubelet\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863966 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-run-ovn-kubernetes\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.863996 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-device-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.864138 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-os-release\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.864146 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b22434e-c80c-469d-9afd-0669319cbf91-os-release\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.864173 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-host-var-lib-cni-multus\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.864208 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-multus-conf-dir\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.864207 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5c1554c-0f80-4c27-bc42-81bb12d90af9-cnibin\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.885862 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" event={"ID":"188ef7b029f22b3628e6f2183332d139","Type":"ContainerStarted","Data":"ebff42c66d5d8f32c82f5101636496f801aa1200132d12cc7539722376131554"} Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.887513 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" event={"ID":"dee390bc22bbe9a006e589c9e7a83c0e","Type":"ContainerStarted","Data":"f079a233df1bd28bde88bd2059c1dce610008c8171864493dd98b8aca6bd490f"} Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.896663 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.900337 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvxq\" (UniqueName: \"kubernetes.io/projected/1b22434e-c80c-469d-9afd-0669319cbf91-kube-api-access-gxvxq\") pod \"multus-additional-cni-plugins-cbh86\" (UID: \"1b22434e-c80c-469d-9afd-0669319cbf91\") " pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.900399 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfldc\" (UniqueName: \"kubernetes.io/projected/d5c1554c-0f80-4c27-bc42-81bb12d90af9-kube-api-access-bfldc\") pod \"multus-tbdks\" (UID: \"d5c1554c-0f80-4c27-bc42-81bb12d90af9\") " pod="openshift-multus/multus-tbdks" Jul 06 19:23:12.910478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.905205 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnzw\" (UniqueName: \"kubernetes.io/projected/e265b652-468c-46c6-8d9e-8c48ae290882-kube-api-access-dmnzw\") pod \"node-ca-59ptz\" (UID: \"e265b652-468c-46c6-8d9e-8c48ae290882\") " pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-systemd\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-etc-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964651 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-kubelet-config\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964729 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-etc-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964750 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-systemd\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-modprobe-d\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964808 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-kubelet-config\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964808 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysconfig\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964838 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovnkube-script-lib\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964856 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysconfig\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-host-slash\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964894 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-host-slash\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964889 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-tmp\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-sys-fs\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.002343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964948 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-modprobe-d\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.964997 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-sys-fs\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965026 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-kubernetes\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-tuned\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965074 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-systemd\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965099 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-var-lib-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965123 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-log-socket\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965145 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-systemd\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965192 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-var-lib-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965234 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-log-socket\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965150 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovnkube-config\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965281 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-kubernetes\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965320 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-etc-selinux\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-kubelet\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965393 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psggn\" (UniqueName: \"kubernetes.io/projected/c5a5d8c4-8f2f-42eb-8113-0002828bee60-kube-api-access-psggn\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.003516 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965417 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovnkube-script-lib\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965422 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-registration-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965474 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-etc-selinux\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965482 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-registration-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965475 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-iptables-alerter-script\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965520 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965545 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-run-netns\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-cni-bin\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-sys\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965624 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovnkube-config\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965624 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-env-overrides\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965665 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3880b-47bb-41bd-a6f6-d2eadcc62027-konnectivity-ca\") pod \"konnectivity-agent-vkf9p\" (UID: \"e7c3880b-47bb-41bd-a6f6-d2eadcc62027\") " pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965685 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysctl-d\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965729 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcczl\" (UniqueName: \"kubernetes.io/projected/97ccad0f-176b-4351-8c9d-25cd80cedd9d-kube-api-access-dcczl\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:13.005318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965754 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovn-node-metrics-cert\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdgm\" (UniqueName: \"kubernetes.io/projected/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-kube-api-access-wmdgm\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965802 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-lib-modules\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965818 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965835 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-ovn\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965866 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysctl-conf\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965924 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-var-lib-kubelet\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-slash\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-cni-netd\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.965987 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248fk\" (UniqueName: \"kubernetes.io/projected/c7ce2adb-2188-42ad-9abb-348cb5444ce9-kube-api-access-248fk\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966000 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-iptables-alerter-script\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966036 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5a5d8c4-8f2f-42eb-8113-0002828bee60-env-overrides\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966059 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-run\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966093 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-kubelet\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.966172 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:13.006204 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-run\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.966249 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret podName:86b026e4-7a94-40ad-bae2-22ae1a89d7a7 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.466211381 +0000 UTC m=+3.108446210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret") pod "global-pull-secret-syncer-6mcrp" (UID: "86b026e4-7a94-40ad-bae2-22ae1a89d7a7") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-host\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966296 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7c3880b-47bb-41bd-a6f6-d2eadcc62027-konnectivity-ca\") pod \"konnectivity-agent-vkf9p\" (UID: \"e7c3880b-47bb-41bd-a6f6-d2eadcc62027\") " pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966326 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-run-netns\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966375 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysctl-d\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966372 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-sys\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966427 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-host\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966472 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-cni-bin\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966501 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-var-lib-kubelet\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966521 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-cni-netd\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966565 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-slash\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966595 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgkr\" (UniqueName: \"kubernetes.io/projected/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-kube-api-access-jrgkr\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966612 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-lib-modules\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.008477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966648 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7c3880b-47bb-41bd-a6f6-d2eadcc62027-agent-certs\") pod \"konnectivity-agent-vkf9p\" (UID: \"e7c3880b-47bb-41bd-a6f6-d2eadcc62027\") " pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966655 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-openvswitch\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966675 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-socket-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966695 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-run-ovn\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.966786 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.966831 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.466816111 +0000 UTC m=+3.109050924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-dbus\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-node-log\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966894 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-run-ovn-kubernetes\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-device-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.966981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-systemd-units\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.009385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967117 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967158 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-node-log\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967199 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-run-ovn-kubernetes\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967244 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-device-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967415 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967442 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5a5d8c4-8f2f-42eb-8113-0002828bee60-systemd-units\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967491 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-sysctl-conf\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967521 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7ce2adb-2188-42ad-9abb-348cb5444ce9-socket-dir\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.967574 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-dbus\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.968507 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-tmp\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.969767 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-etc-tuned\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.970010 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7c3880b-47bb-41bd-a6f6-d2eadcc62027-agent-certs\") pod \"konnectivity-agent-vkf9p\" (UID: \"e7c3880b-47bb-41bd-a6f6-d2eadcc62027\") " pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.970130 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5a5d8c4-8f2f-42eb-8113-0002828bee60-ovn-node-metrics-cert\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.988900 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:13.085360 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.988923 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.988938 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w6ct9 for pod openshift-network-diagnostics/network-check-target-scltl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:12.989282 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9 podName:0e29da91-8c93-4f60-a647-9dbb5920e6c8 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:13.4892633 +0000 UTC m=+3.131498124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w6ct9" (UniqueName: "kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9") pod "network-check-target-scltl" (UID: "0e29da91-8c93-4f60-a647-9dbb5920e6c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.994781 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgkr\" (UniqueName: \"kubernetes.io/projected/3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0-kube-api-access-jrgkr\") pod \"tuned-7jdkc\" (UID: \"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0\") " pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.994883 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdgm\" (UniqueName: \"kubernetes.io/projected/40cbf8dc-0797-4ea1-9e7a-75d2000da1d6-kube-api-access-wmdgm\") pod \"iptables-alerter-8s766\" (UID: \"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6\") " pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.996218 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psggn\" (UniqueName: \"kubernetes.io/projected/c5a5d8c4-8f2f-42eb-8113-0002828bee60-kube-api-access-psggn\") pod \"ovnkube-node-g2flh\" (UID: \"c5a5d8c4-8f2f-42eb-8113-0002828bee60\") " pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.996529 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248fk\" (UniqueName: \"kubernetes.io/projected/c7ce2adb-2188-42ad-9abb-348cb5444ce9-kube-api-access-248fk\") pod \"aws-ebs-csi-driver-node-pv8l4\" (UID: \"c7ce2adb-2188-42ad-9abb-348cb5444ce9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:12.997823 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcczl\" (UniqueName: \"kubernetes.io/projected/97ccad0f-176b-4351-8c9d-25cd80cedd9d-kube-api-access-dcczl\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.059948 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-59ptz" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.067875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cbh86" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.076768 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbdks" Jul 06 19:23:13.086679 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.084693 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8s766" Jul 06 19:23:13.093455 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.092800 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" Jul 06 19:23:13.100698 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.100617 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:13.109818 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.109794 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:13.115558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.115538 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" Jul 06 19:23:13.471269 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.471186 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:13.471269 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.471253 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:13.471492 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.471365 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:13.471492 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.471433 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret podName:86b026e4-7a94-40ad-bae2-22ae1a89d7a7 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:14.471417098 +0000 UTC m=+4.113651903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret") pod "global-pull-secret-syncer-6mcrp" (UID: "86b026e4-7a94-40ad-bae2-22ae1a89d7a7") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:13.471492 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.471372 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:13.471663 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.471529 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:14.471511969 +0000 UTC m=+4.113746774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:13.572119 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.572088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:13.572311 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.572287 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:13.572371 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.572320 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:13.572371 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.572337 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w6ct9 for pod openshift-network-diagnostics/network-check-target-scltl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:13.572491 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.572402 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9 podName:0e29da91-8c93-4f60-a647-9dbb5920e6c8 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:14.572382568 +0000 UTC m=+4.214617387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6ct9" (UniqueName: "kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9") pod "network-check-target-scltl" (UID: "0e29da91-8c93-4f60-a647-9dbb5920e6c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:13.808820 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.808739 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:18:13 +0000 UTC" deadline="2028-01-01 23:08:56.023496878 +0000 UTC" Jul 06 19:23:13.808820 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.808772 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13059h45m42.214728286s" Jul 06 19:23:13.866419 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.866387 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:13.866578 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.866498 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:13.866578 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:13.866563 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:13.866734 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:13.866677 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:14.077873 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:14.077842 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a5d8c4_8f2f_42eb_8113_0002828bee60.slice/crio-6ac8bcc1fde6ddf61d8fa5545bbfe70c018e834d3c28deaeb27ee6d7f098c995 WatchSource:0}: Error finding container 6ac8bcc1fde6ddf61d8fa5545bbfe70c018e834d3c28deaeb27ee6d7f098c995: Status 404 returned error can't find the container with id 6ac8bcc1fde6ddf61d8fa5545bbfe70c018e834d3c28deaeb27ee6d7f098c995 Jul 06 19:23:14.079442 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:14.079417 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b22434e_c80c_469d_9afd_0669319cbf91.slice/crio-c94dac29b3d5a57992e7b333f5d911cf13e5dd4ea80968afb79500d17c191181 WatchSource:0}: Error finding container c94dac29b3d5a57992e7b333f5d911cf13e5dd4ea80968afb79500d17c191181: Status 404 returned error can't find the container with id c94dac29b3d5a57992e7b333f5d911cf13e5dd4ea80968afb79500d17c191181 Jul 06 19:23:14.080307 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:14.080203 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ce2adb_2188_42ad_9abb_348cb5444ce9.slice/crio-c2410356aced7db8a6362bde7c105f15121685bc50b7a240d27c0133648d3d17 WatchSource:0}: Error finding container c2410356aced7db8a6362bde7c105f15121685bc50b7a240d27c0133648d3d17: Status 404 returned error can't find the container with id c2410356aced7db8a6362bde7c105f15121685bc50b7a240d27c0133648d3d17 Jul 06 19:23:14.082183 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:14.081752 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cbf8dc_0797_4ea1_9e7a_75d2000da1d6.slice/crio-6ebbf9ffb6a19cc6dfda8c8e884294f79244c0864ce2c8b5c48f8912c5ea288e WatchSource:0}: Error finding container 6ebbf9ffb6a19cc6dfda8c8e884294f79244c0864ce2c8b5c48f8912c5ea288e: Status 404 returned error can't find the container with id 6ebbf9ffb6a19cc6dfda8c8e884294f79244c0864ce2c8b5c48f8912c5ea288e Jul 06 19:23:14.084102 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:14.084071 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c1554c_0f80_4c27_bc42_81bb12d90af9.slice/crio-83ed47018b279bbbc8e7fe778a6f260e17e8c2482097701e12717f958ce78590 WatchSource:0}: Error finding container 83ed47018b279bbbc8e7fe778a6f260e17e8c2482097701e12717f958ce78590: Status 404 returned error can't find the container with id 83ed47018b279bbbc8e7fe778a6f260e17e8c2482097701e12717f958ce78590 Jul 06 19:23:14.085929 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:14.085819 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c3880b_47bb_41bd_a6f6_d2eadcc62027.slice/crio-73d4653a8e060f43abf40fb813d0f15108ee5e2fe335648f20a6096e3d3f4734 WatchSource:0}: Error finding container 73d4653a8e060f43abf40fb813d0f15108ee5e2fe335648f20a6096e3d3f4734: Status 404 returned error can't find the container with id 73d4653a8e060f43abf40fb813d0f15108ee5e2fe335648f20a6096e3d3f4734 Jul 06 19:23:14.481051 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.480585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:14.481051 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.480892 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:14.481051 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.480928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:14.481051 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.480964 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:16.480942537 +0000 UTC m=+6.123177355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:14.481051 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.481017 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:14.481431 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.481070 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret podName:86b026e4-7a94-40ad-bae2-22ae1a89d7a7 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:16.481054763 +0000 UTC m=+6.123289587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret") pod "global-pull-secret-syncer-6mcrp" (UID: "86b026e4-7a94-40ad-bae2-22ae1a89d7a7") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:14.581981 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.581863 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:14.582131 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.582114 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:14.582201 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.582138 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:14.582201 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.582152 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w6ct9 for pod openshift-network-diagnostics/network-check-target-scltl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:14.582313 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.582212 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9 podName:0e29da91-8c93-4f60-a647-9dbb5920e6c8 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:16.582191613 +0000 UTC m=+6.224426436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6ct9" (UniqueName: "kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9") pod "network-check-target-scltl" (UID: "0e29da91-8c93-4f60-a647-9dbb5920e6c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:14.808973 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.808924 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:18:13 +0000 UTC" deadline="2028-01-12 03:34:49.541078331 +0000 UTC" Jul 06 19:23:14.808973 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.808970 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13304h11m34.732113478s" Jul 06 19:23:14.869154 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.869085 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:14.869306 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:14.869220 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:14.908445 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.908386 2585 generic.go:356] "Generic (PLEG): container finished" podID="dee390bc22bbe9a006e589c9e7a83c0e" containerID="ced39f93d1bda9a11aaa1a96c3779f4e80984f1ae618f50aece22f8ce1454600" exitCode=0 Jul 06 19:23:14.909486 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.909285 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" event={"ID":"dee390bc22bbe9a006e589c9e7a83c0e","Type":"ContainerDied","Data":"ced39f93d1bda9a11aaa1a96c3779f4e80984f1ae618f50aece22f8ce1454600"} Jul 06 19:23:14.921521 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.921433 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-59ptz" event={"ID":"e265b652-468c-46c6-8d9e-8c48ae290882","Type":"ContainerStarted","Data":"4552fed7567944feb22c57d9d098c08fe4dfa1c59612f3b26ecd7af434936255"} Jul 06 19:23:14.929286 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.929222 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8s766" event={"ID":"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6","Type":"ContainerStarted","Data":"6ebbf9ffb6a19cc6dfda8c8e884294f79244c0864ce2c8b5c48f8912c5ea288e"} Jul 06 19:23:14.943904 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.943859 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" event={"ID":"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0","Type":"ContainerStarted","Data":"e68ea341103fb3d5ba36801ae039e803ed967a06edb7a9cdf304b4e992836920"} Jul 06 19:23:14.946946 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.946909 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkf9p" event={"ID":"e7c3880b-47bb-41bd-a6f6-d2eadcc62027","Type":"ContainerStarted","Data":"73d4653a8e060f43abf40fb813d0f15108ee5e2fe335648f20a6096e3d3f4734"} Jul 06 19:23:14.968115 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.968073 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbdks" event={"ID":"d5c1554c-0f80-4c27-bc42-81bb12d90af9","Type":"ContainerStarted","Data":"83ed47018b279bbbc8e7fe778a6f260e17e8c2482097701e12717f958ce78590"} Jul 06 19:23:14.974403 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.974368 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" event={"ID":"c7ce2adb-2188-42ad-9abb-348cb5444ce9","Type":"ContainerStarted","Data":"c2410356aced7db8a6362bde7c105f15121685bc50b7a240d27c0133648d3d17"} Jul 06 19:23:14.996653 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:14.996621 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerStarted","Data":"c94dac29b3d5a57992e7b333f5d911cf13e5dd4ea80968afb79500d17c191181"} Jul 06 19:23:15.018907 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:15.017967 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"6ac8bcc1fde6ddf61d8fa5545bbfe70c018e834d3c28deaeb27ee6d7f098c995"} Jul 06 19:23:15.029485 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:15.029456 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" event={"ID":"188ef7b029f22b3628e6f2183332d139","Type":"ContainerStarted","Data":"01900134f1e5986922e6ed20e17ec8a9cb8fe345330476b2e213a00d66732d2b"} Jul 06 19:23:15.867014 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:15.866985 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:15.867567 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:15.867121 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:15.867567 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:15.867556 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:15.867699 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:15.867673 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:16.051963 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:16.051896 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" event={"ID":"dee390bc22bbe9a006e589c9e7a83c0e","Type":"ContainerStarted","Data":"7fc9aefb17f60440367bea9c89c86714d03adf467e717e9fbb5ca9068e1739d5"} Jul 06 19:23:16.069623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:16.068475 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-106.ec2.internal" podStartSLOduration=4.068454772 podStartE2EDuration="4.068454772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:15.047984166 +0000 UTC m=+4.690218996" watchObservedRunningTime="2026-07-06 19:23:16.068454772 +0000 UTC m=+5.710689600" Jul 06 19:23:16.498644 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:16.498597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:16.498829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:16.498690 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:16.498885 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.498852 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:16.498935 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.498913 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:20.498895665 +0000 UTC m=+10.141130473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:16.499308 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.499288 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:16.499409 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.499341 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret podName:86b026e4-7a94-40ad-bae2-22ae1a89d7a7 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:20.499326516 +0000 UTC m=+10.141561335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret") pod "global-pull-secret-syncer-6mcrp" (UID: "86b026e4-7a94-40ad-bae2-22ae1a89d7a7") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:16.599148 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:16.599115 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:16.599321 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.599309 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:16.599384 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.599330 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:16.599384 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.599352 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w6ct9 for pod openshift-network-diagnostics/network-check-target-scltl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:16.599485 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.599407 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9 podName:0e29da91-8c93-4f60-a647-9dbb5920e6c8 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:20.599388856 +0000 UTC m=+10.241623662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6ct9" (UniqueName: "kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9") pod "network-check-target-scltl" (UID: "0e29da91-8c93-4f60-a647-9dbb5920e6c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:16.869360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:16.868876 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:16.869360 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:16.868994 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:17.867232 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:17.867131 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:17.867419 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:17.867288 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:17.867744 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:17.867598 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:17.867744 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:17.867719 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:18.870684 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:18.868070 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:18.870684 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:18.868276 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:19.866544 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:19.866477 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:19.866735 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:19.866622 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:19.866735 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:19.866478 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:19.866842 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:19.866813 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:20.530816 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:20.530776 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:20.531312 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:20.530843 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:20.531312 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.530971 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:20.531312 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.531013 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:20.531312 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.531053 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:28.531031466 +0000 UTC m=+18.173266271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:20.531312 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.531073 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret podName:86b026e4-7a94-40ad-bae2-22ae1a89d7a7 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:28.53106359 +0000 UTC m=+18.173298399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret") pod "global-pull-secret-syncer-6mcrp" (UID: "86b026e4-7a94-40ad-bae2-22ae1a89d7a7") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:20.632025 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:20.631970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:20.632233 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.632176 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:20.632233 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.632203 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:20.632233 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.632217 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w6ct9 for pod openshift-network-diagnostics/network-check-target-scltl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:20.632394 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.632282 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9 podName:0e29da91-8c93-4f60-a647-9dbb5920e6c8 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:28.632261311 +0000 UTC m=+18.274496119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6ct9" (UniqueName: "kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9") pod "network-check-target-scltl" (UID: "0e29da91-8c93-4f60-a647-9dbb5920e6c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:20.867424 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:20.867346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:20.867569 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:20.867466 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:21.311124 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.310953 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-106.ec2.internal" podStartSLOduration=9.31093474 podStartE2EDuration="9.31093474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:16.069538616 +0000 UTC m=+5.711773443" watchObservedRunningTime="2026-07-06 19:23:21.31093474 +0000 UTC m=+10.953169567" Jul 06 19:23:21.311296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.311190 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4wrf9"] Jul 06 19:23:21.313615 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.313364 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.316165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.315918 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:23:21.316165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.315982 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5d7nv\"" Jul 06 19:23:21.316165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.315925 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:23:21.438825 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.438790 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbs4\" (UniqueName: \"kubernetes.io/projected/1483d7bc-f311-4b6e-857b-14c0189746d9-kube-api-access-swbs4\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.438997 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.438854 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1483d7bc-f311-4b6e-857b-14c0189746d9-hosts-file\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.438997 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.438881 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1483d7bc-f311-4b6e-857b-14c0189746d9-tmp-dir\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.539781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.539744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1483d7bc-f311-4b6e-857b-14c0189746d9-hosts-file\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.540242 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.539807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1483d7bc-f311-4b6e-857b-14c0189746d9-tmp-dir\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.540242 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.539849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbs4\" (UniqueName: \"kubernetes.io/projected/1483d7bc-f311-4b6e-857b-14c0189746d9-kube-api-access-swbs4\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.540242 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.539959 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1483d7bc-f311-4b6e-857b-14c0189746d9-hosts-file\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.540409 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.540291 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1483d7bc-f311-4b6e-857b-14c0189746d9-tmp-dir\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.553441 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.553415 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbs4\" (UniqueName: \"kubernetes.io/projected/1483d7bc-f311-4b6e-857b-14c0189746d9-kube-api-access-swbs4\") pod \"node-resolver-4wrf9\" (UID: \"1483d7bc-f311-4b6e-857b-14c0189746d9\") " pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.626511 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.626438 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wrf9" Jul 06 19:23:21.866754 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.866647 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:21.866898 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:21.866799 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:21.867185 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:21.866647 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:21.867185 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:21.867076 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:22.867623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:22.867301 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:22.867623 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:22.867435 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:23.867184 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:23.867151 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:23.867184 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:23.867170 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:23.867399 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:23.867276 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:23.867475 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:23.867438 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:24.867200 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:24.867166 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:24.867680 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:24.867299 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:25.867387 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:25.867354 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:25.867851 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:25.867749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:25.870111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:25.867898 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:25.870111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:25.868089 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:26.866486 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:26.866456 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:26.866644 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:26.866559 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:27.866443 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:27.866405 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:27.866443 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:27.866441 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:27.866942 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:27.866533 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:27.866942 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:27.866667 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:28.594260 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:28.594224 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:28.594431 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:28.594289 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:28.594431 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.594378 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:28.594431 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.594380 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:28.594555 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.594439 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret podName:86b026e4-7a94-40ad-bae2-22ae1a89d7a7 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:44.594421534 +0000 UTC m=+34.236656376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret") pod "global-pull-secret-syncer-6mcrp" (UID: "86b026e4-7a94-40ad-bae2-22ae1a89d7a7") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:28.594555 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.594460 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:23:44.594449192 +0000 UTC m=+34.236683999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:28.695111 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:28.695073 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:28.695270 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.695203 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:28.695270 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.695219 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:28.695270 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.695230 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w6ct9 for pod openshift-network-diagnostics/network-check-target-scltl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:28.695406 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.695284 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9 podName:0e29da91-8c93-4f60-a647-9dbb5920e6c8 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:44.695268362 +0000 UTC m=+34.337503165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6ct9" (UniqueName: "kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9") pod "network-check-target-scltl" (UID: "0e29da91-8c93-4f60-a647-9dbb5920e6c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:28.866964 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:28.866896 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:28.867378 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:28.867017 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:29.867448 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:29.867415 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:29.867927 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:29.867551 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:29.867927 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:29.867600 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:29.867927 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:29.867732 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:30.867330 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:30.867291 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:30.867507 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:30.867382 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:31.866590 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:31.866557 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:31.866788 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:31.866562 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:31.866788 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:31.866704 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:31.866788 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:31.866758 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:32.433625 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:32.433593 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1483d7bc_f311_4b6e_857b_14c0189746d9.slice/crio-f202b3d39bea0aec68bbcb3b5977f15bd4821ee8413734b0a9fd492f9c549ce7 WatchSource:0}: Error finding container f202b3d39bea0aec68bbcb3b5977f15bd4821ee8413734b0a9fd492f9c549ce7: Status 404 returned error can't find the container with id f202b3d39bea0aec68bbcb3b5977f15bd4821ee8413734b0a9fd492f9c549ce7 Jul 06 19:23:32.867545 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:32.867235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:32.867692 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:32.867649 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:33.084262 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.084235 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" event={"ID":"3831fc90-1fe2-4dd4-a134-8a0fefc6b9c0","Type":"ContainerStarted","Data":"717bd94d2f08fce7aeae93048c752f4da779e1162a3057bc644a24e2ab34735e"} Jul 06 19:23:33.085671 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.085630 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkf9p" event={"ID":"e7c3880b-47bb-41bd-a6f6-d2eadcc62027","Type":"ContainerStarted","Data":"ce5423670fc3b898942a75bd5d88f0ddae5c0c860699a405ddc7ad132a858097"} Jul 06 19:23:33.087060 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.087037 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbdks" event={"ID":"d5c1554c-0f80-4c27-bc42-81bb12d90af9","Type":"ContainerStarted","Data":"146c4238b3156597d26a15ca066cd032d72efb73334862ffe008e22c65e4827a"} Jul 06 19:23:33.088463 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.088445 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" event={"ID":"c7ce2adb-2188-42ad-9abb-348cb5444ce9","Type":"ContainerStarted","Data":"6d9c28020bd90675d31e973aa59977e42dd970b9feed283cb8c1ee6505803883"} Jul 06 19:23:33.089766 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.089743 2585 generic.go:356] "Generic (PLEG): container finished" podID="1b22434e-c80c-469d-9afd-0669319cbf91" containerID="89f4272a0b285f007208cb8807703f69be63c189b069b44f343033e4ac4ae8d1" exitCode=0 Jul 06 19:23:33.089864 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.089833 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerDied","Data":"89f4272a0b285f007208cb8807703f69be63c189b069b44f343033e4ac4ae8d1"} Jul 06 19:23:33.094483 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.094450 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"f5171607394e9f5fcd58ce3a9e83d575a4a51cf06cf2ed70053d7d325a869c0b"} Jul 06 19:23:33.094580 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.094512 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"3f2a5148cbe17a0dd571b009ae985332bca961d874fb97f6b7ae4988f3730f2f"} Jul 06 19:23:33.094580 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.094552 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"c49ae9d24a8dfab84fc90eb840e34c0f77a06a2144e4341f2f995af1ba1c808c"} Jul 06 19:23:33.094725 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.094597 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"dfae97d4f346855964ed0ef2233e8f3ad9711e7c27b818d416bbd5b2e269bdc4"} Jul 06 19:23:33.094725 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.094627 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"0fee0d558d70e384c1db848ef8951bf70f07e43ccd33ae8211d98c15d101596d"} Jul 06 19:23:33.094725 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.094681 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"f69ff65cd3a5622d54bf5486014be2701951327b61e5b227ff1677bae0cbf78b"} Jul 06 19:23:33.095936 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.095917 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-59ptz" event={"ID":"e265b652-468c-46c6-8d9e-8c48ae290882","Type":"ContainerStarted","Data":"c10aed1f118f3b8fce8f41cb8e8ccfa951424e415ebff2e761bf9dbe27d19712"} Jul 06 19:23:33.097348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.097328 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wrf9" event={"ID":"1483d7bc-f311-4b6e-857b-14c0189746d9","Type":"ContainerStarted","Data":"cc6056bf4de61b8f7158cda196008bc104d5744740f8403faa88547f376a5da3"} Jul 06 19:23:33.097467 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.097371 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wrf9" event={"ID":"1483d7bc-f311-4b6e-857b-14c0189746d9","Type":"ContainerStarted","Data":"f202b3d39bea0aec68bbcb3b5977f15bd4821ee8413734b0a9fd492f9c549ce7"} Jul 06 19:23:33.184893 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.184822 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7jdkc" podStartSLOduration=3.835909939 podStartE2EDuration="22.184804393s" totalImagesPullingTime="18.348894454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:33.184607914 +0000 UTC m=+22.826842741" watchObservedRunningTime="2026-07-06 19:23:33.184804393 +0000 UTC m=+22.827039220" Jul 06 19:23:33.333281 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.333232 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vkf9p" podStartSLOduration=12.057213836 podStartE2EDuration="22.333211043s" totalImagesPullingTime="10.275997207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:33.332819985 +0000 UTC m=+22.975054810" watchObservedRunningTime="2026-07-06 19:23:33.333211043 +0000 UTC m=+22.975445870" Jul 06 19:23:33.384365 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.384309 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4wrf9" podStartSLOduration=12.384290034 podStartE2EDuration="12.384290034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:33.383892195 +0000 UTC m=+23.026127023" watchObservedRunningTime="2026-07-06 19:23:33.384290034 +0000 UTC m=+23.026524863" Jul 06 19:23:33.556868 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.556823 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-59ptz" podStartSLOduration=4.21245661 podStartE2EDuration="22.556808392s" totalImagesPullingTime="18.344351782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:33.556745879 +0000 UTC m=+23.198980706" watchObservedRunningTime="2026-07-06 19:23:33.556808392 +0000 UTC m=+23.199043217" Jul 06 19:23:33.752237 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.752193 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tbdks" podStartSLOduration=4.343266972 podStartE2EDuration="22.752179453s" totalImagesPullingTime="18.408912481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:33.751916865 +0000 UTC m=+23.394151691" watchObservedRunningTime="2026-07-06 19:23:33.752179453 +0000 UTC m=+23.394414278" Jul 06 19:23:33.866667 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.866631 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:33.866827 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:33.866804 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:33.866894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.866842 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:33.866949 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:33.866907 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:33.882036 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:33.882014 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:23:34.101756 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.101625 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8s766" event={"ID":"40cbf8dc-0797-4ea1-9e7a-75d2000da1d6","Type":"ContainerStarted","Data":"f7cc8a79099f0ddb1e35f0f99ff24d37eb9df5e57c31cfbec6992604c6ca887d"} Jul 06 19:23:34.103751 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.103692 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" event={"ID":"c7ce2adb-2188-42ad-9abb-348cb5444ce9","Type":"ContainerStarted","Data":"c567633d6d25319718da6d0390045cd25c6f0cd220aeddbff61d58d4436932b1"} Jul 06 19:23:34.136376 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.136326 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8s766" podStartSLOduration=4.784997136 podStartE2EDuration="23.136307078s" totalImagesPullingTime="18.351309942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:34.136161882 +0000 UTC m=+23.778396710" watchObservedRunningTime="2026-07-06 19:23:34.136307078 +0000 UTC m=+23.778541905" Jul 06 19:23:34.621013 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.620979 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:34.621691 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.621643 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:34.857384 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.857285 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:23:33.882033559Z","UUID":"b1c637f6-3056-490a-92f8-ebb89c694a77","Handler":null,"Name":"","Endpoint":""} Jul 06 19:23:34.858797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.858771 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:23:34.858926 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.858812 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:23:34.871612 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:34.871559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:34.871743 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:34.871674 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:35.109371 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:35.109335 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" event={"ID":"c7ce2adb-2188-42ad-9abb-348cb5444ce9","Type":"ContainerStarted","Data":"855bdf767531a1d64f3c9748ce2e21a7e6aad873a040071b95f5bb0237613ede"} Jul 06 19:23:35.109779 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:35.109759 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:35.110262 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:35.110247 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vkf9p" Jul 06 19:23:35.133330 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:35.133291 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pv8l4" podStartSLOduration=3.6528844830000002 podStartE2EDuration="24.133276677s" totalImagesPullingTime="20.480392194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:35.132894139 +0000 UTC m=+24.775128965" watchObservedRunningTime="2026-07-06 19:23:35.133276677 +0000 UTC m=+24.775511502" Jul 06 19:23:35.866844 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:35.866599 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:35.867237 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:35.866642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:35.867237 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:35.866961 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:35.867237 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:35.867029 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:36.115298 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:36.115264 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"967e4d34c1939970a54f183d7f1bc74147fa2d0568344f7aeb11b6952da8f5ff"} Jul 06 19:23:36.867372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:36.867342 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:36.867900 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:36.867452 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:37.867110 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:37.866928 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:37.867326 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:37.866974 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:37.867326 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:37.867188 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:37.867326 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:37.867246 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:38.122599 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.122519 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" event={"ID":"c5a5d8c4-8f2f-42eb-8113-0002828bee60","Type":"ContainerStarted","Data":"b48778e3f91789f5ca3d067d69b84b48a5ff5098927cd6d34b59c6e8fb9e2b7e"} Jul 06 19:23:38.122779 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.122686 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:38.122779 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.122759 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:38.124520 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.124493 2585 generic.go:356] "Generic (PLEG): container finished" podID="1b22434e-c80c-469d-9afd-0669319cbf91" containerID="91587e661c757470befa6730672fa22581b9cd20637696cc7ad200b49f6b89ad" exitCode=0 Jul 06 19:23:38.124616 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.124528 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerDied","Data":"91587e661c757470befa6730672fa22581b9cd20637696cc7ad200b49f6b89ad"} Jul 06 19:23:38.138448 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.138428 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:38.155726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.155691 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" podStartSLOduration=8.734787782 podStartE2EDuration="27.155679s" totalImagesPullingTime="18.420891218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:38.154099887 +0000 UTC m=+27.796334714" watchObservedRunningTime="2026-07-06 19:23:38.155679 +0000 UTC m=+27.797913823" Jul 06 19:23:38.867331 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:38.867226 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:38.867505 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:38.867343 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:39.127707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.127619 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:39.142672 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.142635 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:23:39.426482 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.426459 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4m86h"] Jul 06 19:23:39.426616 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.426592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:39.426718 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:39.426700 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:39.430410 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.430387 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scltl"] Jul 06 19:23:39.430495 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.430477 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:39.430596 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:39.430573 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:39.451050 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.451029 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6mcrp"] Jul 06 19:23:39.451139 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:39.451116 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:39.451219 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:39.451197 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:40.131047 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:40.131013 2585 generic.go:356] "Generic (PLEG): container finished" podID="1b22434e-c80c-469d-9afd-0669319cbf91" containerID="f78d779079afb591b185ced756b51bdd3b59992e380907aacf9648a8ec4535b1" exitCode=0 Jul 06 19:23:40.131472 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:40.131092 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerDied","Data":"f78d779079afb591b185ced756b51bdd3b59992e380907aacf9648a8ec4535b1"} Jul 06 19:23:40.869114 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:40.869089 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:40.869293 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:40.869089 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:40.869293 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:40.869203 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:40.869293 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:40.869273 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:40.869465 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:40.869089 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:40.869465 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:40.869371 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:42.138172 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:42.137981 2585 generic.go:356] "Generic (PLEG): container finished" podID="1b22434e-c80c-469d-9afd-0669319cbf91" containerID="0a43e5b593e5faa90ba117ea92e1f27903e4231683622ca4d971b584e392986b" exitCode=0 Jul 06 19:23:42.138497 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:42.138050 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerDied","Data":"0a43e5b593e5faa90ba117ea92e1f27903e4231683622ca4d971b584e392986b"} Jul 06 19:23:42.867128 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:42.867097 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:42.867128 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:42.867111 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:42.867348 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:42.867098 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:42.867348 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:42.867197 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:42.867348 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:42.867293 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:42.867481 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:42.867396 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:44.613034 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:44.612997 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:44.613637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:44.613058 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:44.613637 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.613161 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:44.613637 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.613242 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret podName:86b026e4-7a94-40ad-bae2-22ae1a89d7a7 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:16.613223098 +0000 UTC m=+66.255457919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret") pod "global-pull-secret-syncer-6mcrp" (UID: "86b026e4-7a94-40ad-bae2-22ae1a89d7a7") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:23:44.613637 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.613160 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:44.613637 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.613316 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:24:16.61330096 +0000 UTC m=+66.255535785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:23:44.714424 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:44.714387 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:44.714586 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.714535 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:23:44.714586 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.714550 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:23:44.714586 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.714563 2585 projected.go:196] Error preparing data for projected volume kube-api-access-w6ct9 for pod openshift-network-diagnostics/network-check-target-scltl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:44.714775 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.714615 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9 podName:0e29da91-8c93-4f60-a647-9dbb5920e6c8 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:16.714597656 +0000 UTC m=+66.356832483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6ct9" (UniqueName: "kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9") pod "network-check-target-scltl" (UID: "0e29da91-8c93-4f60-a647-9dbb5920e6c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:23:44.866792 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:44.866719 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:44.866935 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:44.866731 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:44.866935 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.866848 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6mcrp" podUID="86b026e4-7a94-40ad-bae2-22ae1a89d7a7" Jul 06 19:23:44.866935 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:44.866731 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:44.867094 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.866932 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:23:44.867094 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:44.866976 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scltl" podUID="0e29da91-8c93-4f60-a647-9dbb5920e6c8" Jul 06 19:23:45.139569 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.139502 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-106.ec2.internal" event="NodeReady" Jul 06 19:23:45.139745 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.139645 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:23:45.182989 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.182966 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt"] Jul 06 19:23:45.186090 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.186069 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.188969 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.188788 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 19:23:45.188969 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.188898 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz"] Jul 06 19:23:45.189627 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.189604 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:23:45.190225 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.190045 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:23:45.190225 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.190117 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:23:45.191552 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.191533 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc"] Jul 06 19:23:45.191869 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.191851 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.193920 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.193904 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cc5668599-7qhkt"] Jul 06 19:23:45.194054 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.194032 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:45.194149 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.194052 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 19:23:45.197606 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.196684 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 19:23:45.197606 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.197155 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 19:23:45.197606 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.197228 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 19:23:45.197606 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.197286 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 19:23:45.198145 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.198124 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6bmkh\"" Jul 06 19:23:45.199717 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.199432 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg"] Jul 06 19:23:45.199717 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.199502 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 19:23:45.200017 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.199996 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.202187 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.202172 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt"] Jul 06 19:23:45.202329 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.202316 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbkxs"] Jul 06 19:23:45.202557 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.202537 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5642j\"" Jul 06 19:23:45.202647 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.202475 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:23:45.202740 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.202475 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:23:45.202876 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.202346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.203681 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.203646 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:23:45.206994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.206303 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc"] Jul 06 19:23:45.206994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.206460 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.209473 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.209454 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 19:23:45.209837 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.209821 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xwcb4\"" Jul 06 19:23:45.211991 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.210739 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:23:45.212084 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.210795 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:23:45.212160 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.211426 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fr69w\"" Jul 06 19:23:45.212320 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.211569 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:23:45.212320 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.211885 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz"] Jul 06 19:23:45.212716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.212695 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-227hc"] Jul 06 19:23:45.215161 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.215140 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc5668599-7qhkt"] Jul 06 19:23:45.215241 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.215170 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbkxs"] Jul 06 19:23:45.215241 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.215236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.217765 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.217744 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-96fg5\"" Jul 06 19:23:45.217932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.217915 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:23:45.218163 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.218144 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:23:45.218375 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.218360 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:23:45.225350 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.225329 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg"] Jul 06 19:23:45.226256 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.226235 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-227hc"] Jul 06 19:23:45.319902 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.319876 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fm9h\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-kube-api-access-7fm9h\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320025 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.319910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ca0a1e-d149-4cc9-b1ca-f6ba331280d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg\" (UID: \"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.320025 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.319930 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa6b80e-8165-4bcb-a8bf-306c48480540-tmp-dir\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.320025 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.319958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-image-registry-private-configuration\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320025 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.319985 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320034 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a0256f5-2b8f-4737-96da-772a3f8dc561-cert\") pod \"ingress-canary-227hc\" (UID: \"9a0256f5-2b8f-4737-96da-772a3f8dc561\") " pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.320224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320059 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-trusted-ca\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320109 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9th\" (UniqueName: \"kubernetes.io/projected/9a0256f5-2b8f-4737-96da-772a3f8dc561-kube-api-access-sx9th\") pod \"ingress-canary-227hc\" (UID: \"9a0256f5-2b8f-4737-96da-772a3f8dc561\") " pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.320224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320127 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/309128d8-6d1d-4bd4-95a8-7e460f513d2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.320224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hts\" (UniqueName: \"kubernetes.io/projected/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-kube-api-access-n8hts\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.320224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320201 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa6b80e-8165-4bcb-a8bf-306c48480540-config-volume\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.320224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320223 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z629\" (UniqueName: \"kubernetes.io/projected/309128d8-6d1d-4bd4-95a8-7e460f513d2f-kube-api-access-8z629\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320288 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320316 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320362 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-ca-trust-extracted\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320455 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-bound-sa-token\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320470 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309128d8-6d1d-4bd4-95a8-7e460f513d2f-tmp\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320483 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.320514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c6aae22-4e4c-4149-8495-85a0265dbafd-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:45.320894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320528 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-certificates\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320551 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.320894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320609 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-installation-pull-secrets\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.320894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320645 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-ca\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.320894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320686 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhnt\" (UniqueName: \"kubernetes.io/projected/23ca0a1e-d149-4cc9-b1ca-f6ba331280d3-kube-api-access-qzhnt\") pod \"managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg\" (UID: \"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.320894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320717 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-hub\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.320894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.320762 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2k4\" (UniqueName: \"kubernetes.io/projected/9aa6b80e-8165-4bcb-a8bf-306c48480540-kube-api-access-5h2k4\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.421165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ca0a1e-d149-4cc9-b1ca-f6ba331280d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg\" (UID: \"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.421165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa6b80e-8165-4bcb-a8bf-306c48480540-tmp-dir\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.421165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-image-registry-private-configuration\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421184 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421222 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a0256f5-2b8f-4737-96da-772a3f8dc561-cert\") pod \"ingress-canary-227hc\" (UID: \"9a0256f5-2b8f-4737-96da-772a3f8dc561\") " pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421246 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-trusted-ca\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9th\" (UniqueName: \"kubernetes.io/projected/9a0256f5-2b8f-4737-96da-772a3f8dc561-kube-api-access-sx9th\") pod \"ingress-canary-227hc\" (UID: \"9a0256f5-2b8f-4737-96da-772a3f8dc561\") " pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421313 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/309128d8-6d1d-4bd4-95a8-7e460f513d2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hts\" (UniqueName: \"kubernetes.io/projected/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-kube-api-access-n8hts\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa6b80e-8165-4bcb-a8bf-306c48480540-config-volume\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421394 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z629\" (UniqueName: \"kubernetes.io/projected/309128d8-6d1d-4bd4-95a8-7e460f513d2f-kube-api-access-8z629\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.421421 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421423 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421476 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-ca-trust-extracted\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421563 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-bound-sa-token\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309128d8-6d1d-4bd4-95a8-7e460f513d2f-tmp\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421613 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c6aae22-4e4c-4149-8495-85a0265dbafd-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421680 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-certificates\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421735 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-installation-pull-secrets\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-ca\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421790 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhnt\" (UniqueName: \"kubernetes.io/projected/23ca0a1e-d149-4cc9-b1ca-f6ba331280d3-kube-api-access-qzhnt\") pod \"managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg\" (UID: \"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.421831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-hub\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.422370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2k4\" (UniqueName: \"kubernetes.io/projected/9aa6b80e-8165-4bcb-a8bf-306c48480540-kube-api-access-5h2k4\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.422370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.421903 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fm9h\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-kube-api-access-7fm9h\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.423483 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.423458 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c6aae22-4e4c-4149-8495-85a0265dbafd-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:45.423618 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.423561 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309128d8-6d1d-4bd4-95a8-7e460f513d2f-tmp\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.423771 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.423751 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aa6b80e-8165-4bcb-a8bf-306c48480540-tmp-dir\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.423988 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.423963 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:45.424107 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.424089 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-ca-trust-extracted\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.424196 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.424182 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:45.924156548 +0000 UTC m=+35.566391357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:23:45.425485 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.425010 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:23:45.425485 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.425155 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:45.925134898 +0000 UTC m=+35.567369715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:23:45.425485 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.425195 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.425485 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.425475 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-certificates\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.425894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.425836 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa6b80e-8165-4bcb-a8bf-306c48480540-config-volume\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.425987 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.425922 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:45.425987 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.425940 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:23:45.426078 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.426030 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:45.926010508 +0000 UTC m=+35.568245313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:23:45.426721 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.426645 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-installation-pull-secrets\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.428633 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.428605 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-trusted-ca\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.429963 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.429882 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a0256f5-2b8f-4737-96da-772a3f8dc561-cert\") pod \"ingress-canary-227hc\" (UID: \"9a0256f5-2b8f-4737-96da-772a3f8dc561\") " pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.430056 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.429988 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.430833 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.430163 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-ca\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.430833 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.430717 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-image-registry-private-configuration\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.431144 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.431114 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.431518 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.431482 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ca0a1e-d149-4cc9-b1ca-f6ba331280d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg\" (UID: \"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.432362 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.432311 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-hub\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.432817 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.432793 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/309128d8-6d1d-4bd4-95a8-7e460f513d2f-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.435465 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.435032 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-bound-sa-token\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.435465 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.435150 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2k4\" (UniqueName: \"kubernetes.io/projected/9aa6b80e-8165-4bcb-a8bf-306c48480540-kube-api-access-5h2k4\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.436900 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.436874 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhnt\" (UniqueName: \"kubernetes.io/projected/23ca0a1e-d149-4cc9-b1ca-f6ba331280d3-kube-api-access-qzhnt\") pod \"managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg\" (UID: \"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.437059 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.437036 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z629\" (UniqueName: \"kubernetes.io/projected/309128d8-6d1d-4bd4-95a8-7e460f513d2f-kube-api-access-8z629\") pod \"klusterlet-addon-workmgr-6f5b9d64d-wd5vt\" (UID: \"309128d8-6d1d-4bd4-95a8-7e460f513d2f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.437801 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.437779 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hts\" (UniqueName: \"kubernetes.io/projected/2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f-kube-api-access-n8hts\") pod \"cluster-proxy-proxy-agent-85d4bf67c6-j4jwz\" (UID: \"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.438233 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.438213 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9th\" (UniqueName: \"kubernetes.io/projected/9a0256f5-2b8f-4737-96da-772a3f8dc561-kube-api-access-sx9th\") pod \"ingress-canary-227hc\" (UID: \"9a0256f5-2b8f-4737-96da-772a3f8dc561\") " pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.438629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.438606 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fm9h\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-kube-api-access-7fm9h\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.503271 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.503240 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:45.529186 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.529158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:23:45.548683 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.548644 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:23:45.569879 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.569859 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-227hc" Jul 06 19:23:45.927634 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.927592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:45.927634 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.927640 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:45.927734 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.927779 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.927817 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.927856 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:46.927837045 +0000 UTC m=+36.570071864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.927870 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:46.927864286 +0000 UTC m=+36.570099096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.927868 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.927888 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:23:45.928197 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:45.927944 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:46.927929343 +0000 UTC m=+36.570164149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:23:46.259344 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.259195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt"] Jul 06 19:23:46.262148 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:46.262123 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309128d8_6d1d_4bd4_95a8_7e460f513d2f.slice/crio-4231a8cd913f34120c245f52318345459f16a1373f8d36a74201e49b6f40c442 WatchSource:0}: Error finding container 4231a8cd913f34120c245f52318345459f16a1373f8d36a74201e49b6f40c442: Status 404 returned error can't find the container with id 4231a8cd913f34120c245f52318345459f16a1373f8d36a74201e49b6f40c442 Jul 06 19:23:46.266076 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.266057 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg"] Jul 06 19:23:46.269901 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.268346 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-227hc"] Jul 06 19:23:46.269901 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.269453 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz"] Jul 06 19:23:46.270175 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:46.270149 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23ca0a1e_d149_4cc9_b1ca_f6ba331280d3.slice/crio-30e72dc2896f40bd6fbd449c0ff9af99ca0a1f2a351a9b1b711fd9c155238235 WatchSource:0}: Error finding container 30e72dc2896f40bd6fbd449c0ff9af99ca0a1f2a351a9b1b711fd9c155238235: Status 404 returned error can't find the container with id 30e72dc2896f40bd6fbd449c0ff9af99ca0a1f2a351a9b1b711fd9c155238235 Jul 06 19:23:46.271363 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:46.271339 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0256f5_2b8f_4737_96da_772a3f8dc561.slice/crio-33f60ebf2728011f497405e69058b8455e7677b408878297628bfe5f166d3f5b WatchSource:0}: Error finding container 33f60ebf2728011f497405e69058b8455e7677b408878297628bfe5f166d3f5b: Status 404 returned error can't find the container with id 33f60ebf2728011f497405e69058b8455e7677b408878297628bfe5f166d3f5b Jul 06 19:23:46.272017 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:23:46.271950 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ccd0e1a_a066_4ee4_8fc4_fb622d2dbb4f.slice/crio-5f494341def87f4319bfd708fdc6cb1a8db5f4a677a2c8e8d08a18affe31700a WatchSource:0}: Error finding container 5f494341def87f4319bfd708fdc6cb1a8db5f4a677a2c8e8d08a18affe31700a: Status 404 returned error can't find the container with id 5f494341def87f4319bfd708fdc6cb1a8db5f4a677a2c8e8d08a18affe31700a Jul 06 19:23:46.867037 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.866967 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:23:46.867191 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.866967 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:23:46.867191 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.866968 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:23:46.870128 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.869891 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:23:46.870128 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.869907 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:23:46.871265 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.871237 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5mm8d\"" Jul 06 19:23:46.871372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.871326 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:23:46.871447 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.871427 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:23:46.871558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.871238 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2r6mk\"" Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.945434 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.945484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:46.945598 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:46.945744 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:46.945761 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:46.945814 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:48.945795714 +0000 UTC m=+38.588030522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:46.945883 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:46.945913 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:48.945902647 +0000 UTC m=+38.588137455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:46.945974 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:23:46.946386 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:46.946009 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:48.945995865 +0000 UTC m=+38.588230683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:23:47.155313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:47.155182 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-227hc" event={"ID":"9a0256f5-2b8f-4737-96da-772a3f8dc561","Type":"ContainerStarted","Data":"33f60ebf2728011f497405e69058b8455e7677b408878297628bfe5f166d3f5b"} Jul 06 19:23:47.163171 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:47.163135 2585 generic.go:356] "Generic (PLEG): container finished" podID="1b22434e-c80c-469d-9afd-0669319cbf91" containerID="3a05b80231549ebfcedced4c7587244705476c76253ba1007071ea063b24622a" exitCode=0 Jul 06 19:23:47.163305 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:47.163210 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerDied","Data":"3a05b80231549ebfcedced4c7587244705476c76253ba1007071ea063b24622a"} Jul 06 19:23:47.167819 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:47.167749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" event={"ID":"309128d8-6d1d-4bd4-95a8-7e460f513d2f","Type":"ContainerStarted","Data":"4231a8cd913f34120c245f52318345459f16a1373f8d36a74201e49b6f40c442"} Jul 06 19:23:47.177121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:47.177093 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" event={"ID":"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f","Type":"ContainerStarted","Data":"5f494341def87f4319bfd708fdc6cb1a8db5f4a677a2c8e8d08a18affe31700a"} Jul 06 19:23:47.179484 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:47.179452 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" event={"ID":"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3","Type":"ContainerStarted","Data":"30e72dc2896f40bd6fbd449c0ff9af99ca0a1f2a351a9b1b711fd9c155238235"} Jul 06 19:23:48.195948 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:48.195600 2585 generic.go:356] "Generic (PLEG): container finished" podID="1b22434e-c80c-469d-9afd-0669319cbf91" containerID="da5e3de768d05607716b896216e16e612af8a1365bbf02ce4fda6cd6e123fd45" exitCode=0 Jul 06 19:23:48.197028 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:48.196544 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerDied","Data":"da5e3de768d05607716b896216e16e612af8a1365bbf02ce4fda6cd6e123fd45"} Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:48.965993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:48.966421 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:48.966485 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:48.966669 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:48.966734 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:52.966713888 +0000 UTC m=+42.608948713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:48.967138 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:48.967184 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:23:52.96716918 +0000 UTC m=+42.609404003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:48.967251 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:48.967262 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:23:48.967323 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:48.967291 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:52.967281247 +0000 UTC m=+42.609516067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:23:49.204000 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:49.203935 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cbh86" event={"ID":"1b22434e-c80c-469d-9afd-0669319cbf91","Type":"ContainerStarted","Data":"e843d16e7475b9ed40f67f5576dd6a35d13f91c172471b15c8fec7de09cf6ebc"} Jul 06 19:23:49.233145 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:49.232417 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cbh86" podStartSLOduration=9.670622015 podStartE2EDuration="38.232396797s" totalImagesPullingTime="28.561774782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:49.231261399 +0000 UTC m=+38.873496231" watchObservedRunningTime="2026-07-06 19:23:49.232396797 +0000 UTC m=+38.874631625" Jul 06 19:23:53.000507 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:53.000467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:53.000537 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:53.000576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:53.000641 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:53.000686 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:53.000737 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:53.000754 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:53.000780 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:01.000757756 +0000 UTC m=+50.642992599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:53.000808 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:24:01.00079162 +0000 UTC m=+50.643026453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:23:53.001111 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:23:53.000829 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:01.000819086 +0000 UTC m=+50.643053890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:23:54.224858 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.224818 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-227hc" event={"ID":"9a0256f5-2b8f-4737-96da-772a3f8dc561","Type":"ContainerStarted","Data":"e3654bf3ca212de35fc8e34436059e777b10530c5eee4f6bf6d6f8a97e48af99"} Jul 06 19:23:54.227465 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.227364 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" event={"ID":"309128d8-6d1d-4bd4-95a8-7e460f513d2f","Type":"ContainerStarted","Data":"5a76f0f3c7d825d7f7f8753661ecf7602bb11ac20c0e7854c5d01f686f61c7e0"} Jul 06 19:23:54.227795 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.227639 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:54.229333 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.229112 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:23:54.229439 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.229360 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" event={"ID":"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f","Type":"ContainerStarted","Data":"d9fd1575cc1b2595579fa84c8ddefae482d14bd01eec1457b79a39707795e0f6"} Jul 06 19:23:54.231009 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.230975 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" event={"ID":"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3","Type":"ContainerStarted","Data":"a98a4eef6305cea4920763a1b4c01463c10ead5c5a4be786d4b83f3e8cba9d7e"} Jul 06 19:23:54.242507 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.242413 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-227hc" podStartSLOduration=1.489873109 podStartE2EDuration="9.242399214s" totalImagesPullingTime="7.752526105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:54.241926907 +0000 UTC m=+43.884161735" watchObservedRunningTime="2026-07-06 19:23:54.242399214 +0000 UTC m=+43.884634038" Jul 06 19:23:54.259922 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.259872 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" podStartSLOduration=26.497571859 podStartE2EDuration="34.259854011s" totalImagesPullingTime="7.762282152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:54.259765509 +0000 UTC m=+43.902000358" watchObservedRunningTime="2026-07-06 19:23:54.259854011 +0000 UTC m=+43.902088838" Jul 06 19:23:54.278293 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:54.278254 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" podStartSLOduration=26.525252905 podStartE2EDuration="34.278242474s" totalImagesPullingTime="7.752989569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:54.277384897 +0000 UTC m=+43.919619724" watchObservedRunningTime="2026-07-06 19:23:54.278242474 +0000 UTC m=+43.920477300" Jul 06 19:23:55.238530 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:55.238493 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" event={"ID":"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f","Type":"ContainerStarted","Data":"157425cb7d53193422714746b0edada6e8ddd97a689c2740247d625c2067cc10"} Jul 06 19:23:55.238915 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:55.238537 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" event={"ID":"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f","Type":"ContainerStarted","Data":"cc0bec303a37844ab134cfce2ced231f319261a31b31e2287eb1baa1ecb3f5ab"} Jul 06 19:23:55.260512 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:23:55.260470 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" podStartSLOduration=27.494114461 podStartE2EDuration="35.260457354s" totalImagesPullingTime="7.766342893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:55.259299372 +0000 UTC m=+44.901534188" watchObservedRunningTime="2026-07-06 19:23:55.260457354 +0000 UTC m=+44.902692184" Jul 06 19:24:01.062883 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:01.062845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:24:01.062883 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:01.062888 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:01.062946 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:01.063003 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:01.063035 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:01.063043 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:01.063055 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:01.063094 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:24:17.063073438 +0000 UTC m=+66.705308245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:01.063112 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:17.063103097 +0000 UTC m=+66.705337914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:24:01.063288 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:01.063125 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:17.063117823 +0000 UTC m=+66.705352642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:24:11.144004 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:11.143978 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g2flh" Jul 06 19:24:16.683990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.683952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:24:16.684530 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.684050 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:24:16.686963 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.686943 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:24:16.687064 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.687051 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:24:16.694995 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:16.694976 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:24:16.695071 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:16.695031 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:25:20.695015894 +0000 UTC m=+130.337250698 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : secret "metrics-daemon-secret" not found Jul 06 19:24:16.698728 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.698710 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/86b026e4-7a94-40ad-bae2-22ae1a89d7a7-original-pull-secret\") pod \"global-pull-secret-syncer-6mcrp\" (UID: \"86b026e4-7a94-40ad-bae2-22ae1a89d7a7\") " pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:24:16.785021 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.784987 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:24:16.787609 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.787590 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:24:16.797965 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.797948 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:24:16.808387 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.808362 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ct9\" (UniqueName: \"kubernetes.io/projected/0e29da91-8c93-4f60-a647-9dbb5920e6c8-kube-api-access-w6ct9\") pod \"network-check-target-scltl\" (UID: \"0e29da91-8c93-4f60-a647-9dbb5920e6c8\") " pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:24:16.895559 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.895532 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2r6mk\"" Jul 06 19:24:16.900695 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.900680 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6mcrp" Jul 06 19:24:16.903801 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:16.903785 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:24:17.064483 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:17.064454 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6mcrp"] Jul 06 19:24:17.067607 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:24:17.067581 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b026e4_7a94_40ad_bae2_22ae1a89d7a7.slice/crio-1279e018333ef9a5f0140a8a42259b11f76af8a3e4aa501e2873e6c920f42f07 WatchSource:0}: Error finding container 1279e018333ef9a5f0140a8a42259b11f76af8a3e4aa501e2873e6c920f42f07: Status 404 returned error can't find the container with id 1279e018333ef9a5f0140a8a42259b11f76af8a3e4aa501e2873e6c920f42f07 Jul 06 19:24:17.081725 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:17.081706 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scltl"] Jul 06 19:24:17.083851 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:24:17.083829 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e29da91_8c93_4f60_a647_9dbb5920e6c8.slice/crio-48d78ad0773bee891d0356f148117a6873b1da0c7979dcbfcd8a41ad6bec7ae7 WatchSource:0}: Error finding container 48d78ad0773bee891d0356f148117a6873b1da0c7979dcbfcd8a41ad6bec7ae7: Status 404 returned error can't find the container with id 48d78ad0773bee891d0356f148117a6873b1da0c7979dcbfcd8a41ad6bec7ae7 Jul 06 19:24:17.087771 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:17.087752 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:24:17.087845 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:17.087783 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:24:17.087845 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:17.087827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:24:17.087912 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:17.087893 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:24:17.087912 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:17.087904 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:24:17.087974 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:17.087913 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:24:17.087974 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:17.087934 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:24:17.087974 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:17.087942 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:49.087928006 +0000 UTC m=+98.730162823 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:24:17.087974 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:17.087956 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:49.087949658 +0000 UTC m=+98.730184462 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:24:17.088109 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:17.087977 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:24:49.087961672 +0000 UTC m=+98.730196476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:24:17.300344 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:17.300305 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scltl" event={"ID":"0e29da91-8c93-4f60-a647-9dbb5920e6c8","Type":"ContainerStarted","Data":"48d78ad0773bee891d0356f148117a6873b1da0c7979dcbfcd8a41ad6bec7ae7"} Jul 06 19:24:17.301215 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:17.301189 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6mcrp" event={"ID":"86b026e4-7a94-40ad-bae2-22ae1a89d7a7","Type":"ContainerStarted","Data":"1279e018333ef9a5f0140a8a42259b11f76af8a3e4aa501e2873e6c920f42f07"} Jul 06 19:24:22.319960 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:22.319921 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6mcrp" event={"ID":"86b026e4-7a94-40ad-bae2-22ae1a89d7a7","Type":"ContainerStarted","Data":"c7560e4bdcc0b471de92f26a32cf9ea94764bb923bc93d935c76b3b186073bb2"} Jul 06 19:24:22.321405 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:22.321382 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scltl" event={"ID":"0e29da91-8c93-4f60-a647-9dbb5920e6c8","Type":"ContainerStarted","Data":"775b8d54d89afb7e4d1c1ad75279e36012e7478865552f2a8ac034595048bd84"} Jul 06 19:24:22.321528 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:22.321514 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:24:22.341630 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:22.341587 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6mcrp" podStartSLOduration=66.833008413 podStartE2EDuration="1m11.341573806s" totalImagesPullingTime="4.508565393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:22.340878277 +0000 UTC m=+71.983113110" watchObservedRunningTime="2026-07-06 19:24:22.341573806 +0000 UTC m=+71.983808633" Jul 06 19:24:22.356869 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:22.356833 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-scltl" podStartSLOduration=66.867611027 podStartE2EDuration="1m11.356821185s" totalImagesPullingTime="4.489210158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:22.355916834 +0000 UTC m=+71.998151662" watchObservedRunningTime="2026-07-06 19:24:22.356821185 +0000 UTC m=+71.999055989" Jul 06 19:24:49.135305 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:49.135279 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:49.135346 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:49.135378 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:49.135457 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:49.135480 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cc5668599-7qhkt: secret "image-registry-tls" not found Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:49.135487 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:49.135491 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:49.135548 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls podName:8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98 nodeName:}" failed. No retries permitted until 2026-07-06 19:25:53.135528541 +0000 UTC m=+162.777763346 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls") pod "image-registry-cc5668599-7qhkt" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98") : secret "image-registry-tls" not found Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:49.135571 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls podName:9aa6b80e-8165-4bcb-a8bf-306c48480540 nodeName:}" failed. No retries permitted until 2026-07-06 19:25:53.135559114 +0000 UTC m=+162.777793924 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls") pod "dns-default-dbkxs" (UID: "9aa6b80e-8165-4bcb-a8bf-306c48480540") : secret "dns-default-metrics-tls" not found Jul 06 19:24:49.135702 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:24:49.135587 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:25:53.135578083 +0000 UTC m=+162.777812887 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:24:53.331045 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:24:53.331017 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-scltl" Jul 06 19:25:20.760783 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:20.760736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:25:20.761277 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:20.760885 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:25:20.761277 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:20.760965 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs podName:97ccad0f-176b-4351-8c9d-25cd80cedd9d nodeName:}" failed. No retries permitted until 2026-07-06 19:27:22.760948331 +0000 UTC m=+252.403183135 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs") pod "network-metrics-daemon-4m86h" (UID: "97ccad0f-176b-4351-8c9d-25cd80cedd9d") : secret "metrics-daemon-secret" not found Jul 06 19:25:34.874160 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.874130 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-drf7c"] Jul 06 19:25:34.876123 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.876108 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:34.879006 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.878975 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:25:34.879145 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.879006 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:25:34.879145 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.879009 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:25:34.879145 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.879065 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:25:34.880262 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.880242 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hv5fq\"" Jul 06 19:25:34.908776 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.908754 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-drf7c"] Jul 06 19:25:34.960648 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.960612 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74cae87-3597-419b-803c-d0e4b93c861e-data-volume\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:34.960648 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.960644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:34.960893 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.960727 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:34.960893 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.960765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtrw\" (UniqueName: \"kubernetes.io/projected/b74cae87-3597-419b-803c-d0e4b93c861e-kube-api-access-rqtrw\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:34.960893 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:34.960818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b74cae87-3597-419b-803c-d0e4b93c861e-crio-socket\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.061363 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.061325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74cae87-3597-419b-803c-d0e4b93c861e-data-volume\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.061363 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.061360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.061389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.061414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtrw\" (UniqueName: \"kubernetes.io/projected/b74cae87-3597-419b-803c-d0e4b93c861e-kube-api-access-rqtrw\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.061471 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.061515 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.061533 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b74cae87-3597-419b-803c-d0e4b93c861e-crio-socket\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.061552 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:35.561530181 +0000 UTC m=+145.203765008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : configmap "kube-rbac-proxy" not found Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.061581 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:35.561564683 +0000 UTC m=+145.203799487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : secret "insights-runtime-extractor-tls" not found Jul 06 19:25:35.061594 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.061477 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b74cae87-3597-419b-803c-d0e4b93c861e-crio-socket\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.061897 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.061769 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74cae87-3597-419b-803c-d0e4b93c861e-data-volume\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.070925 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.070907 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtrw\" (UniqueName: \"kubernetes.io/projected/b74cae87-3597-419b-803c-d0e4b93c861e-kube-api-access-rqtrw\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.564239 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.564213 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.564409 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:35.564257 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:35.564409 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.564334 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:25:35.564409 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.564377 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:25:35.564409 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.564392 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:36.564376347 +0000 UTC m=+146.206611151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : configmap "kube-rbac-proxy" not found Jul 06 19:25:35.564544 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:35.564426 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:36.564415662 +0000 UTC m=+146.206650466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : secret "insights-runtime-extractor-tls" not found Jul 06 19:25:36.571785 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:36.571730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:36.572213 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:36.571838 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:36.572213 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:36.571878 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:25:36.572213 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:36.571901 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:25:36.572213 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:36.571938 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:38.57192364 +0000 UTC m=+148.214158443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : secret "insights-runtime-extractor-tls" not found Jul 06 19:25:36.572213 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:36.571952 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:38.571946203 +0000 UTC m=+148.214181007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : configmap "kube-rbac-proxy" not found Jul 06 19:25:36.664109 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:36.664082 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wrf9_1483d7bc-f311-4b6e-857b-14c0189746d9/dns-node-resolver/0.log" Jul 06 19:25:37.665304 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:37.665265 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-59ptz_e265b652-468c-46c6-8d9e-8c48ae290882/node-ca/0.log" Jul 06 19:25:38.470742 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:38.470715 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-227hc_9a0256f5-2b8f-4737-96da-772a3f8dc561/serve-healthcheck-canary/0.log" Jul 06 19:25:38.587832 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:38.587791 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:38.588023 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:38.587855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:38.588023 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:38.587910 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:25:38.588023 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:38.587971 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:25:38.588023 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:38.587983 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:42.587967171 +0000 UTC m=+152.230201975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : configmap "kube-rbac-proxy" not found Jul 06 19:25:38.588023 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:38.588020 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:42.588004925 +0000 UTC m=+152.230239743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : secret "insights-runtime-extractor-tls" not found Jul 06 19:25:42.620028 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:42.619974 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:42.620557 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:42.620049 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:42.620557 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:42.620103 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:25:42.620557 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:42.620166 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:25:42.620557 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:42.620182 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:50.620164486 +0000 UTC m=+160.262399294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : configmap "kube-rbac-proxy" not found Jul 06 19:25:42.620557 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:42.620234 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:25:50.620218319 +0000 UTC m=+160.262453140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : secret "insights-runtime-extractor-tls" not found Jul 06 19:25:48.235131 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:48.235088 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" podUID="9c6aae22-4e4c-4149-8495-85a0265dbafd" Jul 06 19:25:48.242220 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:48.242194 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" podUID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" Jul 06 19:25:48.263440 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:48.263409 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dbkxs" podUID="9aa6b80e-8165-4bcb-a8bf-306c48480540" Jul 06 19:25:48.576716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:48.576628 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbkxs" Jul 06 19:25:48.576849 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:48.576628 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:25:48.576849 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:48.576632 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:25:49.882882 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:49.882835 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4m86h" podUID="97ccad0f-176b-4351-8c9d-25cd80cedd9d" Jul 06 19:25:50.682259 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:50.682229 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:50.682422 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:50.682270 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:50.682422 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:50.682380 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:25:50.682493 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:50.682442 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls podName:b74cae87-3597-419b-803c-d0e4b93c861e nodeName:}" failed. No retries permitted until 2026-07-06 19:26:06.682427738 +0000 UTC m=+176.324662542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-drf7c" (UID: "b74cae87-3597-419b-803c-d0e4b93c861e") : secret "insights-runtime-extractor-tls" not found Jul 06 19:25:50.682803 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:50.682784 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b74cae87-3597-419b-803c-d0e4b93c861e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:25:53.201211 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.201173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:25:53.201721 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.201225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:25:53.201721 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.201250 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:25:53.201721 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:53.201349 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:25:53.201721 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:25:53.201405 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert podName:9c6aae22-4e4c-4149-8495-85a0265dbafd nodeName:}" failed. No retries permitted until 2026-07-06 19:27:55.201388554 +0000 UTC m=+284.843623359 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-z6dqc" (UID: "9c6aae22-4e4c-4149-8495-85a0265dbafd") : secret "networking-console-plugin-cert" not found Jul 06 19:25:53.203455 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.203430 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"image-registry-cc5668599-7qhkt\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:25:53.203541 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.203464 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9aa6b80e-8165-4bcb-a8bf-306c48480540-metrics-tls\") pod \"dns-default-dbkxs\" (UID: \"9aa6b80e-8165-4bcb-a8bf-306c48480540\") " pod="openshift-dns/dns-default-dbkxs" Jul 06 19:25:53.379869 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.379835 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fr69w\"" Jul 06 19:25:53.380700 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.380684 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5642j\"" Jul 06 19:25:53.387741 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.387722 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:25:53.387837 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.387801 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbkxs" Jul 06 19:25:53.522014 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.521990 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbkxs"] Jul 06 19:25:53.524225 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:25:53.524197 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa6b80e_8165_4bcb_a8bf_306c48480540.slice/crio-9f6e23c74a16a0f7c7527e9e75ebcc3396ede0ae30746cf97a24bd1367e3de15 WatchSource:0}: Error finding container 9f6e23c74a16a0f7c7527e9e75ebcc3396ede0ae30746cf97a24bd1367e3de15: Status 404 returned error can't find the container with id 9f6e23c74a16a0f7c7527e9e75ebcc3396ede0ae30746cf97a24bd1367e3de15 Jul 06 19:25:53.538521 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.538499 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc5668599-7qhkt"] Jul 06 19:25:53.540712 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:25:53.540689 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dea61d3_90cd_4f6c_b3f1_a6dedb9f1f98.slice/crio-cffe2e20f1bf731401e1586b11ddfa92bdbe7041057f86deea1b68da9c57dae9 WatchSource:0}: Error finding container cffe2e20f1bf731401e1586b11ddfa92bdbe7041057f86deea1b68da9c57dae9: Status 404 returned error can't find the container with id cffe2e20f1bf731401e1586b11ddfa92bdbe7041057f86deea1b68da9c57dae9 Jul 06 19:25:53.592497 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.592472 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" event={"ID":"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98","Type":"ContainerStarted","Data":"cffe2e20f1bf731401e1586b11ddfa92bdbe7041057f86deea1b68da9c57dae9"} Jul 06 19:25:53.593588 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:53.593567 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbkxs" event={"ID":"9aa6b80e-8165-4bcb-a8bf-306c48480540","Type":"ContainerStarted","Data":"9f6e23c74a16a0f7c7527e9e75ebcc3396ede0ae30746cf97a24bd1367e3de15"} Jul 06 19:25:54.228269 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.228201 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" podUID="309128d8-6d1d-4bd4-95a8-7e460f513d2f" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.6:8000/readyz\": dial tcp 10.132.0.6:8000: connect: connection refused" Jul 06 19:25:54.598186 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.598151 2585 generic.go:356] "Generic (PLEG): container finished" podID="309128d8-6d1d-4bd4-95a8-7e460f513d2f" containerID="5a76f0f3c7d825d7f7f8753661ecf7602bb11ac20c0e7854c5d01f686f61c7e0" exitCode=1 Jul 06 19:25:54.598428 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.598237 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" event={"ID":"309128d8-6d1d-4bd4-95a8-7e460f513d2f","Type":"ContainerDied","Data":"5a76f0f3c7d825d7f7f8753661ecf7602bb11ac20c0e7854c5d01f686f61c7e0"} Jul 06 19:25:54.598611 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.598596 2585 scope.go:117] "RemoveContainer" containerID="5a76f0f3c7d825d7f7f8753661ecf7602bb11ac20c0e7854c5d01f686f61c7e0" Jul 06 19:25:54.599993 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.599871 2585 generic.go:356] "Generic (PLEG): container finished" podID="23ca0a1e-d149-4cc9-b1ca-f6ba331280d3" containerID="a98a4eef6305cea4920763a1b4c01463c10ead5c5a4be786d4b83f3e8cba9d7e" exitCode=255 Jul 06 19:25:54.599993 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.599918 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" event={"ID":"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3","Type":"ContainerDied","Data":"a98a4eef6305cea4920763a1b4c01463c10ead5c5a4be786d4b83f3e8cba9d7e"} Jul 06 19:25:54.600236 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.600219 2585 scope.go:117] "RemoveContainer" containerID="a98a4eef6305cea4920763a1b4c01463c10ead5c5a4be786d4b83f3e8cba9d7e" Jul 06 19:25:54.601757 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.601730 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" event={"ID":"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98","Type":"ContainerStarted","Data":"388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3"} Jul 06 19:25:54.601894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.601878 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:25:54.664395 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:54.664348 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" podStartSLOduration=163.664331167 podStartE2EDuration="2m43.664331167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:54.663764694 +0000 UTC m=+164.305999523" watchObservedRunningTime="2026-07-06 19:25:54.664331167 +0000 UTC m=+164.306565990" Jul 06 19:25:55.504163 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:55.504120 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:25:55.549141 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:55.549107 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" Jul 06 19:25:55.606630 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:55.606600 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" event={"ID":"309128d8-6d1d-4bd4-95a8-7e460f513d2f","Type":"ContainerStarted","Data":"b941cda671a98f92a8c185de113657dae31462c5876d80d17eacff55cc578115"} Jul 06 19:25:55.606874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:55.606852 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:25:55.607848 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:55.607820 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9d64d-wd5vt" Jul 06 19:25:55.608499 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:55.608470 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54b9f7c64b-rlsdg" event={"ID":"23ca0a1e-d149-4cc9-b1ca-f6ba331280d3","Type":"ContainerStarted","Data":"d6c3c5059593fd9451c370b3ddf34372dc4668e01348c262198d285107169d61"} Jul 06 19:25:56.613249 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:56.613212 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbkxs" event={"ID":"9aa6b80e-8165-4bcb-a8bf-306c48480540","Type":"ContainerStarted","Data":"6803e8104d6ede5770a4d9eb92b4dcf9a10bcb1d8f1faff0f2df74a924df1e96"} Jul 06 19:25:56.613618 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:56.613302 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbkxs" event={"ID":"9aa6b80e-8165-4bcb-a8bf-306c48480540","Type":"ContainerStarted","Data":"78655c4b405dba4440e971b5b7d7a84e975d16f0ab3bd6afda9221747ec748b0"} Jul 06 19:25:56.634503 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:56.634462 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dbkxs" podStartSLOduration=129.562285901 podStartE2EDuration="2m11.634449097s" totalImagesPullingTime="2.072163196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:56.63407082 +0000 UTC m=+166.276305647" watchObservedRunningTime="2026-07-06 19:25:56.634449097 +0000 UTC m=+166.276683923" Jul 06 19:25:57.617300 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:25:57.617265 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbkxs" Jul 06 19:26:04.867213 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:04.867137 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:26:06.703617 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:06.703579 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:26:06.705894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:06.705874 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b74cae87-3597-419b-803c-d0e4b93c861e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drf7c\" (UID: \"b74cae87-3597-419b-803c-d0e4b93c861e\") " pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:26:06.984522 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:06.984494 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-drf7c" Jul 06 19:26:07.118257 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:07.118086 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-drf7c"] Jul 06 19:26:07.120720 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:26:07.120694 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74cae87_3597_419b_803c_d0e4b93c861e.slice/crio-f3919fb83261e7676df42eef75f285845b40c842a6ec9cf1d4793e4ffb9e7525 WatchSource:0}: Error finding container f3919fb83261e7676df42eef75f285845b40c842a6ec9cf1d4793e4ffb9e7525: Status 404 returned error can't find the container with id f3919fb83261e7676df42eef75f285845b40c842a6ec9cf1d4793e4ffb9e7525 Jul 06 19:26:07.623050 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:07.623017 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbkxs" Jul 06 19:26:07.652942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:07.652902 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drf7c" event={"ID":"b74cae87-3597-419b-803c-d0e4b93c861e","Type":"ContainerStarted","Data":"f3919fb83261e7676df42eef75f285845b40c842a6ec9cf1d4793e4ffb9e7525"} Jul 06 19:26:11.672761 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:11.672727 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drf7c" event={"ID":"b74cae87-3597-419b-803c-d0e4b93c861e","Type":"ContainerStarted","Data":"020774e36c96bec70131a479f9d1eb5bdc6c6da985cbc564b5632b1a63b1dc25"} Jul 06 19:26:11.673103 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:11.672804 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drf7c" event={"ID":"b74cae87-3597-419b-803c-d0e4b93c861e","Type":"ContainerStarted","Data":"9f696a30797f87ecf88e3ab96cc23f8ac0d598d8bf831bc6d7903cd73335580f"} Jul 06 19:26:13.392929 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:13.392888 2585 patch_prober.go:28] interesting pod/image-registry-cc5668599-7qhkt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 19:26:13.393482 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:13.392953 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" podUID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:26:14.685430 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:14.685394 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drf7c" event={"ID":"b74cae87-3597-419b-803c-d0e4b93c861e","Type":"ContainerStarted","Data":"072d582a2b23379c09c336e891076178a8fc0713b18a5e7c7346c71266168ab6"} Jul 06 19:26:14.707224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:14.707172 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-drf7c" podStartSLOduration=33.858375988 podStartE2EDuration="40.70715756s" totalImagesPullingTime="6.848781572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:25:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:14.705964423 +0000 UTC m=+184.348199250" watchObservedRunningTime="2026-07-06 19:26:14.70715756 +0000 UTC m=+184.349392386" Jul 06 19:26:15.612945 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:15.612917 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:26:20.280708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.280676 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wjbtw"] Jul 06 19:26:20.283872 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.283855 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.286507 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.286474 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:26:20.286687 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.286557 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:26:20.286687 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.286557 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:26:20.286687 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.286576 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:26:20.287775 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.287752 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sgsxh\"" Jul 06 19:26:20.287899 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.287755 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:26:20.287899 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.287779 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:26:20.415178 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-tls\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415371 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415205 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-sys\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415371 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-root\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415371 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85bt\" (UniqueName: \"kubernetes.io/projected/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-kube-api-access-s85bt\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415371 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-metrics-client-ca\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415371 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415348 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415373 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-wtmp\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.415558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.415415 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-textfile\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516149 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516116 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-textfile\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516289 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-tls\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516289 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-sys\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516289 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-root\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516289 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85bt\" (UniqueName: \"kubernetes.io/projected/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-kube-api-access-s85bt\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516289 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-metrics-client-ca\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516292 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-sys\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516328 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-root\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:26:20.516360 2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516388 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:26:20.516416 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-tls podName:6cc288b3-1a2c-452d-8ff6-44e9d4db2d19 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:21.016398285 +0000 UTC m=+190.658633092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-tls") pod "node-exporter-wjbtw" (UID: "6cc288b3-1a2c-452d-8ff6-44e9d4db2d19") : secret "node-exporter-tls" not found Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516486 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-textfile\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.516532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-wtmp\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.517006 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516684 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-wtmp\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.517006 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516861 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-metrics-client-ca\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.517006 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.516919 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.518760 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.518742 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:20.529140 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:20.529117 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85bt\" (UniqueName: \"kubernetes.io/projected/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-kube-api-access-s85bt\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:21.020513 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:21.020472 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-tls\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:21.022847 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:21.022822 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6cc288b3-1a2c-452d-8ff6-44e9d4db2d19-node-exporter-tls\") pod \"node-exporter-wjbtw\" (UID: \"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19\") " pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:21.193680 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:21.193629 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wjbtw" Jul 06 19:26:21.206292 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:26:21.206264 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc288b3_1a2c_452d_8ff6_44e9d4db2d19.slice/crio-838b9a2cefc53c766e21bd0d978ecdf6f1b9371c9e5e92dbd97866662a03648d WatchSource:0}: Error finding container 838b9a2cefc53c766e21bd0d978ecdf6f1b9371c9e5e92dbd97866662a03648d: Status 404 returned error can't find the container with id 838b9a2cefc53c766e21bd0d978ecdf6f1b9371c9e5e92dbd97866662a03648d Jul 06 19:26:21.708370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:21.708324 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjbtw" event={"ID":"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19","Type":"ContainerStarted","Data":"838b9a2cefc53c766e21bd0d978ecdf6f1b9371c9e5e92dbd97866662a03648d"} Jul 06 19:26:22.713481 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:22.713450 2585 generic.go:356] "Generic (PLEG): container finished" podID="6cc288b3-1a2c-452d-8ff6-44e9d4db2d19" containerID="5e2453f99178696fa71f9e1f945afc0be39db3e51e0640ca0380b6f91dceef32" exitCode=0 Jul 06 19:26:22.713867 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:22.713517 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjbtw" event={"ID":"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19","Type":"ContainerDied","Data":"5e2453f99178696fa71f9e1f945afc0be39db3e51e0640ca0380b6f91dceef32"} Jul 06 19:26:23.718526 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:23.718479 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjbtw" event={"ID":"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19","Type":"ContainerStarted","Data":"afdfb604688b83755d4990d188a6ad0a29be35d4d03685576d4bea75a4df9e92"} Jul 06 19:26:23.718953 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:23.718548 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjbtw" event={"ID":"6cc288b3-1a2c-452d-8ff6-44e9d4db2d19","Type":"ContainerStarted","Data":"b85c938882fdf87642375f0eb40efdafd6eea9b7d919aefecddb29c1d93b0ded"} Jul 06 19:26:23.742533 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:23.742480 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wjbtw" podStartSLOduration=2.864320496 podStartE2EDuration="3.742464695s" totalImagesPullingTime="815.914135ms" totalInitContainerRuntime="62.230064ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:23.741190124 +0000 UTC m=+193.383424950" watchObservedRunningTime="2026-07-06 19:26:23.742464695 +0000 UTC m=+193.384699521" Jul 06 19:26:25.530733 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:25.530696 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" podUID="2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:26:32.893515 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:32.893477 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc5668599-7qhkt"] Jul 06 19:26:35.530876 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:35.530834 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" podUID="2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:26:45.530755 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:45.530716 2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" podUID="2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:26:45.531201 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:45.530783 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" Jul 06 19:26:45.531285 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:45.531246 2585 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"157425cb7d53193422714746b0edada6e8ddd97a689c2740247d625c2067cc10"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 06 19:26:45.531321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:45.531307 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" podUID="2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f" containerName="service-proxy" containerID="cri-o://157425cb7d53193422714746b0edada6e8ddd97a689c2740247d625c2067cc10" gracePeriod=30 Jul 06 19:26:45.794673 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:45.794577 2585 generic.go:356] "Generic (PLEG): container finished" podID="2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f" containerID="157425cb7d53193422714746b0edada6e8ddd97a689c2740247d625c2067cc10" exitCode=2 Jul 06 19:26:45.794673 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:45.794635 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" event={"ID":"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f","Type":"ContainerDied","Data":"157425cb7d53193422714746b0edada6e8ddd97a689c2740247d625c2067cc10"} Jul 06 19:26:45.794849 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:45.794684 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85d4bf67c6-j4jwz" event={"ID":"2ccd0e1a-a066-4ee4-8fc4-fb622d2dbb4f","Type":"ContainerStarted","Data":"e44c38500f86379d2d9372f53b3154faf3268a4d41ac7157d59f47e5fdf5304a"} Jul 06 19:26:57.912567 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:57.912511 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" podUID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" containerName="registry" containerID="cri-o://388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3" gracePeriod=30 Jul 06 19:26:58.170926 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.170863 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:26:58.199794 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.199765 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-certificates\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.199975 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.199804 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-trusted-ca\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.199975 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.199828 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-bound-sa-token\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.199975 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.199861 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.199975 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.199901 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-ca-trust-extracted\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.199975 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.199930 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-installation-pull-secrets\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.199975 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.199969 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fm9h\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-kube-api-access-7fm9h\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.200272 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.200024 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-image-registry-private-configuration\") pod \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\" (UID: \"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98\") " Jul 06 19:26:58.200272 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.200200 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:26:58.200272 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.200257 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-certificates\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.201165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.201132 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:26:58.203720 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.203672 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:58.203838 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.203730 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:26:58.203921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.203852 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-kube-api-access-7fm9h" (OuterVolumeSpecName: "kube-api-access-7fm9h") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "kube-api-access-7fm9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:58.203921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.203858 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:26:58.204170 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.204150 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:58.210510 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.210483 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" (UID: "8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:26:58.300994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.300949 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-trusted-ca\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.300994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.300985 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-bound-sa-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.300994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.300997 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-registry-tls\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.301231 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.301007 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-ca-trust-extracted\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.301231 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.301023 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-installation-pull-secrets\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.301231 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.301035 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fm9h\" (UniqueName: \"kubernetes.io/projected/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-kube-api-access-7fm9h\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.301231 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.301048 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98-image-registry-private-configuration\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:26:58.838817 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.838778 2585 generic.go:356] "Generic (PLEG): container finished" podID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" containerID="388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3" exitCode=0 Jul 06 19:26:58.838997 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.838825 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" Jul 06 19:26:58.838997 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.838861 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" event={"ID":"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98","Type":"ContainerDied","Data":"388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3"} Jul 06 19:26:58.838997 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.838914 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc5668599-7qhkt" event={"ID":"8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98","Type":"ContainerDied","Data":"cffe2e20f1bf731401e1586b11ddfa92bdbe7041057f86deea1b68da9c57dae9"} Jul 06 19:26:58.838997 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.838934 2585 scope.go:117] "RemoveContainer" containerID="388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3" Jul 06 19:26:58.847911 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.847893 2585 scope.go:117] "RemoveContainer" containerID="388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3" Jul 06 19:26:58.848175 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:26:58.848156 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3\": container with ID starting with 388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3 not found: ID does not exist" containerID="388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3" Jul 06 19:26:58.848230 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.848183 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3"} err="failed to get container status \"388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3\": rpc error: code = NotFound desc = could not find container \"388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3\": container with ID starting with 388f2008c8beec2f8b3b6d21b7969e7b9fa65e8bf10086fc0bb2a8d4d2faf8e3 not found: ID does not exist" Jul 06 19:26:58.860919 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.860888 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cc5668599-7qhkt"] Jul 06 19:26:58.863928 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.863901 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cc5668599-7qhkt"] Jul 06 19:26:58.871186 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:26:58.871161 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" path="/var/lib/kubelet/pods/8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98/volumes" Jul 06 19:27:22.786211 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:22.786173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:27:22.788410 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:22.788389 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97ccad0f-176b-4351-8c9d-25cd80cedd9d-metrics-certs\") pod \"network-metrics-daemon-4m86h\" (UID: \"97ccad0f-176b-4351-8c9d-25cd80cedd9d\") " pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:27:22.870670 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:22.870638 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5mm8d\"" Jul 06 19:27:22.879173 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:22.879157 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4m86h" Jul 06 19:27:23.007179 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:23.007158 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4m86h"] Jul 06 19:27:23.009477 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:27:23.009446 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ccad0f_176b_4351_8c9d_25cd80cedd9d.slice/crio-067bdbe1cf59438891c05b2bfc54163118ba79b0f552513d7d5937f55f5e2dbf WatchSource:0}: Error finding container 067bdbe1cf59438891c05b2bfc54163118ba79b0f552513d7d5937f55f5e2dbf: Status 404 returned error can't find the container with id 067bdbe1cf59438891c05b2bfc54163118ba79b0f552513d7d5937f55f5e2dbf Jul 06 19:27:23.920213 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:23.920179 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4m86h" event={"ID":"97ccad0f-176b-4351-8c9d-25cd80cedd9d","Type":"ContainerStarted","Data":"067bdbe1cf59438891c05b2bfc54163118ba79b0f552513d7d5937f55f5e2dbf"} Jul 06 19:27:24.924271 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:24.924238 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4m86h" event={"ID":"97ccad0f-176b-4351-8c9d-25cd80cedd9d","Type":"ContainerStarted","Data":"ba8f99babbc86c0046a6d749cceca3c25ce7ec08ee4ca23ddeca3ff0e3089d54"} Jul 06 19:27:24.924675 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:24.924290 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4m86h" event={"ID":"97ccad0f-176b-4351-8c9d-25cd80cedd9d","Type":"ContainerStarted","Data":"8292930cc61730b3131cc2f77fd1b4a4c1e5c1157086ada608d5905a1275deb0"} Jul 06 19:27:24.942816 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:24.942766 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4m86h" podStartSLOduration=252.553566094 podStartE2EDuration="4m13.942751579s" totalImagesPullingTime="1.389185485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:24.942416055 +0000 UTC m=+254.584650882" watchObservedRunningTime="2026-07-06 19:27:24.942751579 +0000 UTC m=+254.584986405" Jul 06 19:27:51.577232 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:27:51.577173 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" podUID="9c6aae22-4e4c-4149-8495-85a0265dbafd" Jul 06 19:27:52.013425 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:52.013389 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:27:55.236411 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:55.236363 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:27:55.238900 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:55.238874 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c6aae22-4e4c-4149-8495-85a0265dbafd-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-z6dqc\" (UID: \"9c6aae22-4e4c-4149-8495-85a0265dbafd\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:27:55.317418 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:55.317390 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6bmkh\"" Jul 06 19:27:55.325542 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:55.325520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" Jul 06 19:27:55.457966 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:55.454098 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc"] Jul 06 19:27:56.027670 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:56.027629 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" event={"ID":"9c6aae22-4e4c-4149-8495-85a0265dbafd","Type":"ContainerStarted","Data":"017b36dd089c4f7af570f1a96acc1f24608e229bd4fd415ff5ae721b357a649e"} Jul 06 19:27:57.033366 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:57.033320 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" event={"ID":"9c6aae22-4e4c-4149-8495-85a0265dbafd","Type":"ContainerStarted","Data":"9e0212523579b10f3d777333fd9f438b2beae071b6ca1b371e2ca8bc2cf86fc3"} Jul 06 19:27:57.050880 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:27:57.050833 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-z6dqc" podStartSLOduration=273.814167726 podStartE2EDuration="4m35.050819041s" totalImagesPullingTime="1.236651315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:57.050145509 +0000 UTC m=+286.692380334" watchObservedRunningTime="2026-07-06 19:27:57.050819041 +0000 UTC m=+286.693053906" Jul 06 19:28:10.764809 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:28:10.764780 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:29:11.856613 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.856529 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb"] Jul 06 19:29:11.857126 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.856919 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" containerName="registry" Jul 06 19:29:11.857126 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.856937 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" containerName="registry" Jul 06 19:29:11.857126 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.856991 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dea61d3-90cd-4f6c-b3f1-a6dedb9f1f98" containerName="registry" Jul 06 19:29:11.859815 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.859794 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:11.862317 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.862297 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:29:11.862413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.862371 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:29:11.863496 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.863480 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mbpbc\"" Jul 06 19:29:11.872642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.872620 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb"] Jul 06 19:29:11.938341 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.938296 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/281e044c-1d92-454b-b8ac-e92ba4a6ae46-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hvpjb\" (UID: \"281e044c-1d92-454b-b8ac-e92ba4a6ae46\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:11.938520 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:11.938350 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzk8x\" (UniqueName: \"kubernetes.io/projected/281e044c-1d92-454b-b8ac-e92ba4a6ae46-kube-api-access-lzk8x\") pod \"cert-manager-cainjector-559cdd8c87-hvpjb\" (UID: \"281e044c-1d92-454b-b8ac-e92ba4a6ae46\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:12.038941 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:12.038897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzk8x\" (UniqueName: \"kubernetes.io/projected/281e044c-1d92-454b-b8ac-e92ba4a6ae46-kube-api-access-lzk8x\") pod \"cert-manager-cainjector-559cdd8c87-hvpjb\" (UID: \"281e044c-1d92-454b-b8ac-e92ba4a6ae46\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:12.039113 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:12.038952 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/281e044c-1d92-454b-b8ac-e92ba4a6ae46-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hvpjb\" (UID: \"281e044c-1d92-454b-b8ac-e92ba4a6ae46\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:12.049545 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:12.049518 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/281e044c-1d92-454b-b8ac-e92ba4a6ae46-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hvpjb\" (UID: \"281e044c-1d92-454b-b8ac-e92ba4a6ae46\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:12.049722 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:12.049703 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzk8x\" (UniqueName: \"kubernetes.io/projected/281e044c-1d92-454b-b8ac-e92ba4a6ae46-kube-api-access-lzk8x\") pod \"cert-manager-cainjector-559cdd8c87-hvpjb\" (UID: \"281e044c-1d92-454b-b8ac-e92ba4a6ae46\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:12.168914 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:12.168832 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" Jul 06 19:29:12.298627 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:12.298599 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb"] Jul 06 19:29:12.301589 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:29:12.301562 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod281e044c_1d92_454b_b8ac_e92ba4a6ae46.slice/crio-8cf36799414640d2770dcfbbcd07e00f4bfe768eeb4778f55c2c9ce065f63ca3 WatchSource:0}: Error finding container 8cf36799414640d2770dcfbbcd07e00f4bfe768eeb4778f55c2c9ce065f63ca3: Status 404 returned error can't find the container with id 8cf36799414640d2770dcfbbcd07e00f4bfe768eeb4778f55c2c9ce065f63ca3 Jul 06 19:29:12.303400 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:12.303383 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:29:13.281817 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:13.281768 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" event={"ID":"281e044c-1d92-454b-b8ac-e92ba4a6ae46","Type":"ContainerStarted","Data":"8cf36799414640d2770dcfbbcd07e00f4bfe768eeb4778f55c2c9ce065f63ca3"} Jul 06 19:29:16.293048 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:16.293016 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" event={"ID":"281e044c-1d92-454b-b8ac-e92ba4a6ae46","Type":"ContainerStarted","Data":"8ee0733c7b38dabfceddb4331d46bc002d8f3b6a6dcafa354b3d704eebfe3f46"} Jul 06 19:29:16.315637 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:16.315587 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hvpjb" podStartSLOduration=2.3589366 podStartE2EDuration="5.315570441s" totalImagesPullingTime="2.956633841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:16.314580711 +0000 UTC m=+365.956815540" watchObservedRunningTime="2026-07-06 19:29:16.315570441 +0000 UTC m=+365.957805267" Jul 06 19:29:43.650862 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.650828 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl"] Jul 06 19:29:43.653926 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.653908 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.656793 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.656773 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bkc2r\"" Jul 06 19:29:43.656909 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.656819 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:29:43.656973 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.656938 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:29:43.657153 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.657140 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:29:43.657383 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.657370 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:29:43.671722 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.671699 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl"] Jul 06 19:29:43.763672 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.763623 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d423-34ef-4155-90b0-5244db05d22f-tmp\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.763672 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.763674 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2c5d423-34ef-4155-90b0-5244db05d22f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.763878 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.763708 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d423-34ef-4155-90b0-5244db05d22f-manifests\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.763878 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.763760 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7xv\" (UniqueName: \"kubernetes.io/projected/f2c5d423-34ef-4155-90b0-5244db05d22f-kube-api-access-kw7xv\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.763878 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.763798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2c5d423-34ef-4155-90b0-5244db05d22f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.864557 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.864527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d423-34ef-4155-90b0-5244db05d22f-tmp\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.864557 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.864559 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2c5d423-34ef-4155-90b0-5244db05d22f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.864771 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.864586 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d423-34ef-4155-90b0-5244db05d22f-manifests\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.864771 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.864611 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7xv\" (UniqueName: \"kubernetes.io/projected/f2c5d423-34ef-4155-90b0-5244db05d22f-kube-api-access-kw7xv\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.864771 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.864638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2c5d423-34ef-4155-90b0-5244db05d22f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.865070 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.865036 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d423-34ef-4155-90b0-5244db05d22f-manifests\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.866839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.866820 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c5d423-34ef-4155-90b0-5244db05d22f-tmp\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.867225 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.867203 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2c5d423-34ef-4155-90b0-5244db05d22f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.867281 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.867264 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2c5d423-34ef-4155-90b0-5244db05d22f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.878415 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.878389 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7xv\" (UniqueName: \"kubernetes.io/projected/f2c5d423-34ef-4155-90b0-5244db05d22f-kube-api-access-kw7xv\") pod \"opendatahub-operator-controller-manager-7fcb8c7879-p8wwl\" (UID: \"f2c5d423-34ef-4155-90b0-5244db05d22f\") " pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:43.963308 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:43.963224 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:44.094558 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:44.094536 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl"] Jul 06 19:29:44.097029 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:29:44.096998 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c5d423_34ef_4155_90b0_5244db05d22f.slice/crio-f6456f88d63b40d9ad1a5b661d85f6e45f87b888f831a2b2bb0a4f3e447cf9ed WatchSource:0}: Error finding container f6456f88d63b40d9ad1a5b661d85f6e45f87b888f831a2b2bb0a4f3e447cf9ed: Status 404 returned error can't find the container with id f6456f88d63b40d9ad1a5b661d85f6e45f87b888f831a2b2bb0a4f3e447cf9ed Jul 06 19:29:44.391251 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:44.391216 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" event={"ID":"f2c5d423-34ef-4155-90b0-5244db05d22f","Type":"ContainerStarted","Data":"f6456f88d63b40d9ad1a5b661d85f6e45f87b888f831a2b2bb0a4f3e447cf9ed"} Jul 06 19:29:47.402642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:47.402569 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" event={"ID":"f2c5d423-34ef-4155-90b0-5244db05d22f","Type":"ContainerStarted","Data":"862572ae08c03d5dbadfcc97af32d62615c486d0afa40bce5adca4ab552d333d"} Jul 06 19:29:48.407898 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:48.407785 2585 generic.go:356] "Generic (PLEG): container finished" podID="f2c5d423-34ef-4155-90b0-5244db05d22f" containerID="862572ae08c03d5dbadfcc97af32d62615c486d0afa40bce5adca4ab552d333d" exitCode=0 Jul 06 19:29:48.407898 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:48.407846 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" event={"ID":"f2c5d423-34ef-4155-90b0-5244db05d22f","Type":"ContainerDied","Data":"862572ae08c03d5dbadfcc97af32d62615c486d0afa40bce5adca4ab552d333d"} Jul 06 19:29:49.413642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:49.413604 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" event={"ID":"f2c5d423-34ef-4155-90b0-5244db05d22f","Type":"ContainerStarted","Data":"3e3965ee91ef09ae1b43ac2094ec24e8dc61f60fd553dac2c412d7f04bc74c76"} Jul 06 19:29:49.414039 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:49.413724 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:29:49.436572 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:29:49.436513 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" podStartSLOduration=1.917346159 podStartE2EDuration="6.436491825s" totalImagesPullingTime="3.499640537s" totalInitContainerRuntime="1.019505129s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:29:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:49.435558559 +0000 UTC m=+399.077793389" watchObservedRunningTime="2026-07-06 19:29:49.436491825 +0000 UTC m=+399.078726652" Jul 06 19:30:00.419114 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:00.419079 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fcb8c7879-p8wwl" Jul 06 19:30:13.662560 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.662520 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb"] Jul 06 19:30:13.667476 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.667455 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.670475 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.670446 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:30:13.670619 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.670443 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:30:13.670619 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.670486 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:30:13.671566 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.671546 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:30:13.671676 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.671573 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gs9s4\"" Jul 06 19:30:13.674019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.673924 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtsk\" (UniqueName: \"kubernetes.io/projected/b2095723-c283-4e1b-a9ce-4893b588b5b9-kube-api-access-cmtsk\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.674019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.673961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2095723-c283-4e1b-a9ce-4893b588b5b9-tmp\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.674019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.673994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2095723-c283-4e1b-a9ce-4893b588b5b9-tls-certs\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.687490 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.687463 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb"] Jul 06 19:30:13.774427 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.774390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtsk\" (UniqueName: \"kubernetes.io/projected/b2095723-c283-4e1b-a9ce-4893b588b5b9-kube-api-access-cmtsk\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.774427 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.774435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2095723-c283-4e1b-a9ce-4893b588b5b9-tmp\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.774642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.774545 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2095723-c283-4e1b-a9ce-4893b588b5b9-tls-certs\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.776668 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.776624 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2095723-c283-4e1b-a9ce-4893b588b5b9-tmp\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.776971 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.776953 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2095723-c283-4e1b-a9ce-4893b588b5b9-tls-certs\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.795740 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.795717 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mrp5j"] Jul 06 19:30:13.798759 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.798741 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:13.799414 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.799397 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtsk\" (UniqueName: \"kubernetes.io/projected/b2095723-c283-4e1b-a9ce-4893b588b5b9-kube-api-access-cmtsk\") pod \"kube-auth-proxy-54bf5b454b-q6pwb\" (UID: \"b2095723-c283-4e1b-a9ce-4893b588b5b9\") " pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:13.801062 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.801041 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 19:30:13.801159 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.801051 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z2qjs\"" Jul 06 19:30:13.810593 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.810549 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mrp5j"] Jul 06 19:30:13.874921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.874880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgks\" (UniqueName: \"kubernetes.io/projected/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-kube-api-access-9pgks\") pod \"odh-model-controller-7998c67dc4-mrp5j\" (UID: \"d6c1bf2d-7302-4f07-99bb-db4a88b6351b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:13.875092 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.874946 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-cert\") pod \"odh-model-controller-7998c67dc4-mrp5j\" (UID: \"d6c1bf2d-7302-4f07-99bb-db4a88b6351b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:13.975861 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.975777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgks\" (UniqueName: \"kubernetes.io/projected/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-kube-api-access-9pgks\") pod \"odh-model-controller-7998c67dc4-mrp5j\" (UID: \"d6c1bf2d-7302-4f07-99bb-db4a88b6351b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:13.975861 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.975831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-cert\") pod \"odh-model-controller-7998c67dc4-mrp5j\" (UID: \"d6c1bf2d-7302-4f07-99bb-db4a88b6351b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:13.976049 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:30:13.975923 2585 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 19:30:13.976049 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:30:13.975974 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-cert podName:d6c1bf2d-7302-4f07-99bb-db4a88b6351b nodeName:}" failed. No retries permitted until 2026-07-06 19:30:14.475959528 +0000 UTC m=+424.118194332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-cert") pod "odh-model-controller-7998c67dc4-mrp5j" (UID: "d6c1bf2d-7302-4f07-99bb-db4a88b6351b") : secret "odh-model-controller-webhook-cert" not found Jul 06 19:30:13.976243 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:13.976226 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" Jul 06 19:30:14.005924 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:14.005892 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgks\" (UniqueName: \"kubernetes.io/projected/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-kube-api-access-9pgks\") pod \"odh-model-controller-7998c67dc4-mrp5j\" (UID: \"d6c1bf2d-7302-4f07-99bb-db4a88b6351b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:14.125827 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:14.125804 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb"] Jul 06 19:30:14.128146 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:30:14.128125 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2095723_c283_4e1b_a9ce_4893b588b5b9.slice/crio-da886689087a8d5fc0a304d9b4ddb9f0624317f05cfd763c9db0e123098d05e3 WatchSource:0}: Error finding container da886689087a8d5fc0a304d9b4ddb9f0624317f05cfd763c9db0e123098d05e3: Status 404 returned error can't find the container with id da886689087a8d5fc0a304d9b4ddb9f0624317f05cfd763c9db0e123098d05e3 Jul 06 19:30:14.479183 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:14.479147 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-cert\") pod \"odh-model-controller-7998c67dc4-mrp5j\" (UID: \"d6c1bf2d-7302-4f07-99bb-db4a88b6351b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:14.481482 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:14.481456 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6c1bf2d-7302-4f07-99bb-db4a88b6351b-cert\") pod \"odh-model-controller-7998c67dc4-mrp5j\" (UID: \"d6c1bf2d-7302-4f07-99bb-db4a88b6351b\") " pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:14.501597 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:14.501569 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" event={"ID":"b2095723-c283-4e1b-a9ce-4893b588b5b9","Type":"ContainerStarted","Data":"da886689087a8d5fc0a304d9b4ddb9f0624317f05cfd763c9db0e123098d05e3"} Jul 06 19:30:14.714746 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:14.714713 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:14.878429 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:14.878378 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mrp5j"] Jul 06 19:30:15.506193 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:15.506155 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" event={"ID":"d6c1bf2d-7302-4f07-99bb-db4a88b6351b","Type":"ContainerStarted","Data":"233f532e957e04024ff605cc1c1e3e7459b062256de0dcbe3db61a6bb26652e2"} Jul 06 19:30:18.519778 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:18.519743 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" event={"ID":"b2095723-c283-4e1b-a9ce-4893b588b5b9","Type":"ContainerStarted","Data":"b51e2383bc70a3c5b7873c2481cce48177dfe0bd88cdfd2c6b33eed0c544c2c2"} Jul 06 19:30:18.521211 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:18.521180 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" event={"ID":"d6c1bf2d-7302-4f07-99bb-db4a88b6351b","Type":"ContainerStarted","Data":"ca85f8744c3b06a289dfe104d9e1b9c061bf0969b7288a2330c5a61d62748256"} Jul 06 19:30:18.521356 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:18.521246 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:18.544069 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:18.543979 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54bf5b454b-q6pwb" podStartSLOduration=1.393153443 podStartE2EDuration="5.543962507s" totalImagesPullingTime="4.150809064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:18.542212561 +0000 UTC m=+428.184447386" watchObservedRunningTime="2026-07-06 19:30:18.543962507 +0000 UTC m=+428.186197338" Jul 06 19:30:18.563292 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:18.563215 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" podStartSLOduration=2.157479708 podStartE2EDuration="5.563189234s" totalImagesPullingTime="3.405709526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:18.561753587 +0000 UTC m=+428.203988414" watchObservedRunningTime="2026-07-06 19:30:18.563189234 +0000 UTC m=+428.205424063" Jul 06 19:30:20.387085 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.387055 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs"] Jul 06 19:30:20.394055 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.394033 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.398532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.398509 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g5vg4\"" Jul 06 19:30:20.399358 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.399335 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:30:20.422083 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422042 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422230 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2183335f-beb5-42ef-8199-137449dfb291-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422230 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422128 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422230 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7mb\" (UniqueName: \"kubernetes.io/projected/2183335f-beb5-42ef-8199-137449dfb291-kube-api-access-mk7mb\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422404 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422231 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2183335f-beb5-42ef-8199-137449dfb291-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422404 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-istio-data\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422404 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422311 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2183335f-beb5-42ef-8199-137449dfb291-istio-token\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422404 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422379 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.422591 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.422434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.423646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.423628 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs"] Jul 06 19:30:20.522895 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.522855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2183335f-beb5-42ef-8199-137449dfb291-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.522895 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.522900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-istio-data\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523150 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.522922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2183335f-beb5-42ef-8199-137449dfb291-istio-token\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523150 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.522950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523150 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.522989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523336 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523189 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523336 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523240 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2183335f-beb5-42ef-8199-137449dfb291-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523336 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523276 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523336 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523295 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7mb\" (UniqueName: \"kubernetes.io/projected/2183335f-beb5-42ef-8199-137449dfb291-kube-api-access-mk7mb\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523556 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523356 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-istio-data\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523556 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523445 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523556 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523448 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523556 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523539 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.523822 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.523701 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2183335f-beb5-42ef-8199-137449dfb291-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.525457 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.525431 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2183335f-beb5-42ef-8199-137449dfb291-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.525812 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.525794 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2183335f-beb5-42ef-8199-137449dfb291-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.533368 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.533345 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2183335f-beb5-42ef-8199-137449dfb291-istio-token\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.533631 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.533614 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7mb\" (UniqueName: \"kubernetes.io/projected/2183335f-beb5-42ef-8199-137449dfb291-kube-api-access-mk7mb\") pod \"data-science-gateway-data-science-gateway-class-568d56697dbk4zs\" (UID: \"2183335f-beb5-42ef-8199-137449dfb291\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.703782 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.703695 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:20.859042 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:20.859000 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs"] Jul 06 19:30:20.863119 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:30:20.863084 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2183335f_beb5_42ef_8199_137449dfb291.slice/crio-3b6c8d17b5633ae6e3ecaa75f0805bc87f8d04a3aef57600a76b25bd5efff43d WatchSource:0}: Error finding container 3b6c8d17b5633ae6e3ecaa75f0805bc87f8d04a3aef57600a76b25bd5efff43d: Status 404 returned error can't find the container with id 3b6c8d17b5633ae6e3ecaa75f0805bc87f8d04a3aef57600a76b25bd5efff43d Jul 06 19:30:21.536734 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:21.536627 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" event={"ID":"2183335f-beb5-42ef-8199-137449dfb291","Type":"ContainerStarted","Data":"3b6c8d17b5633ae6e3ecaa75f0805bc87f8d04a3aef57600a76b25bd5efff43d"} Jul 06 19:30:22.406203 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.406165 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t"] Jul 06 19:30:22.408246 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.408224 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.412245 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.412048 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:30:22.412245 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.412071 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:30:22.412245 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.412110 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-s6bll\"" Jul 06 19:30:22.412245 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.412215 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:30:22.412510 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.412382 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:30:22.412722 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.412700 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:30:22.423196 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.423174 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t"] Jul 06 19:30:22.438740 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.438709 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4466f\" (UniqueName: \"kubernetes.io/projected/948b1022-c227-4641-bb08-3b955d81702b-kube-api-access-4466f\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.438865 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.438755 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/948b1022-c227-4641-bb08-3b955d81702b-cert\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.438865 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.438781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/948b1022-c227-4641-bb08-3b955d81702b-manager-config\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.438865 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.438857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/948b1022-c227-4641-bb08-3b955d81702b-metrics-cert\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.539843 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.539798 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4466f\" (UniqueName: \"kubernetes.io/projected/948b1022-c227-4641-bb08-3b955d81702b-kube-api-access-4466f\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.539843 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.539842 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/948b1022-c227-4641-bb08-3b955d81702b-cert\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.540416 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.539867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/948b1022-c227-4641-bb08-3b955d81702b-manager-config\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.540416 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.539893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/948b1022-c227-4641-bb08-3b955d81702b-metrics-cert\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.540812 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.540784 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/948b1022-c227-4641-bb08-3b955d81702b-manager-config\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.543004 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.542979 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/948b1022-c227-4641-bb08-3b955d81702b-cert\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.543132 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.543087 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/948b1022-c227-4641-bb08-3b955d81702b-metrics-cert\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.555584 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.555559 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4466f\" (UniqueName: \"kubernetes.io/projected/948b1022-c227-4641-bb08-3b955d81702b-kube-api-access-4466f\") pod \"lws-controller-manager-cf88bd784-bl46t\" (UID: \"948b1022-c227-4641-bb08-3b955d81702b\") " pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:22.720497 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:22.720400 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:23.464907 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:23.464825 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:30:23.465030 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:23.464941 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:30:23.465030 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:23.464988 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:30:23.546360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:23.546230 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t"] Jul 06 19:30:23.548295 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:30:23.548265 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948b1022_c227_4641_bb08_3b955d81702b.slice/crio-1fbc032a6dce0432cedd8cb49913c1f259c00a3bedacedeaf315149dd27abd62 WatchSource:0}: Error finding container 1fbc032a6dce0432cedd8cb49913c1f259c00a3bedacedeaf315149dd27abd62: Status 404 returned error can't find the container with id 1fbc032a6dce0432cedd8cb49913c1f259c00a3bedacedeaf315149dd27abd62 Jul 06 19:30:23.549198 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:23.549165 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" event={"ID":"2183335f-beb5-42ef-8199-137449dfb291","Type":"ContainerStarted","Data":"0f416b6df404555cec66bbac3e983f418c60f6a7a8d9450360d5ebbe1b3e7627"} Jul 06 19:30:23.703984 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:23.703886 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:24.556724 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:24.556685 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" event={"ID":"948b1022-c227-4641-bb08-3b955d81702b","Type":"ContainerStarted","Data":"1fbc032a6dce0432cedd8cb49913c1f259c00a3bedacedeaf315149dd27abd62"} Jul 06 19:30:24.709989 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:24.709958 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:24.732821 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:24.732771 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" podStartSLOduration=2.133748534 podStartE2EDuration="4.732753407s" totalImagesPullingTime="2.599004873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:23.578631705 +0000 UTC m=+433.220866532" watchObservedRunningTime="2026-07-06 19:30:24.732753407 +0000 UTC m=+434.374988232" Jul 06 19:30:25.561489 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:25.561436 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:25.562678 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:25.562629 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-568d56697dbk4zs" Jul 06 19:30:26.566227 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:26.566188 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" event={"ID":"948b1022-c227-4641-bb08-3b955d81702b","Type":"ContainerStarted","Data":"c80ece02c25253b38b7995a92653d6478cffe8431cc3854d931207fdfe5bf17c"} Jul 06 19:30:26.589551 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:26.589499 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" podStartSLOduration=2.241263516 podStartE2EDuration="4.589481577s" totalImagesPullingTime="2.348218061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:26.588674072 +0000 UTC m=+436.230908890" watchObservedRunningTime="2026-07-06 19:30:26.589481577 +0000 UTC m=+436.231716402" Jul 06 19:30:27.570479 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:27.570448 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:30:29.527208 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:29.527178 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:30:38.576505 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:30:38.576426 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cf88bd784-bl46t" Jul 06 19:32:46.675049 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.675010 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j"] Jul 06 19:32:46.677232 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.677216 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.679935 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.679916 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jbh6w\"" Jul 06 19:32:46.691502 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.691480 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j"] Jul 06 19:32:46.747530 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747503 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-data\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747565 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-workload-certs\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747602 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-credential-socket\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747646 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-token\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747790 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr99c\" (UniqueName: \"kubernetes.io/projected/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-kube-api-access-cr99c\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.747874 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.747844 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-workload-socket\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.848955 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.848925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849080 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.848966 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-token\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849080 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.848993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr99c\" (UniqueName: \"kubernetes.io/projected/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-kube-api-access-cr99c\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849080 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-workload-socket\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849080 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849286 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849085 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-data\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849286 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-workload-certs\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849391 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849311 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-credential-socket\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849391 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849524 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849503 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-data\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849742 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849721 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-workload-socket\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849876 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849750 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-workload-certs\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.849876 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849789 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-credential-socket\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.850021 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.849943 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.851435 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.851413 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.851435 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.851424 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.857523 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.857495 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-istio-token\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.857960 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.857941 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr99c\" (UniqueName: \"kubernetes.io/projected/8a8f7a8d-d28e-45e5-be42-3137ba6e139b-kube-api-access-cr99c\") pod \"maas-default-gateway-openshift-default-5dfc694f5c-cdq4j\" (UID: \"8a8f7a8d-d28e-45e5-be42-3137ba6e139b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:46.986686 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:46.986643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:47.120621 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:47.120580 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j"] Jul 06 19:32:47.125912 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:32:47.125879 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8f7a8d_d28e_45e5_be42_3137ba6e139b.slice/crio-2d7b80980682d90ad34b99903ca4234e6dbbb527027e9c58f8238ebd2cfc989e WatchSource:0}: Error finding container 2d7b80980682d90ad34b99903ca4234e6dbbb527027e9c58f8238ebd2cfc989e: Status 404 returned error can't find the container with id 2d7b80980682d90ad34b99903ca4234e6dbbb527027e9c58f8238ebd2cfc989e Jul 06 19:32:47.127908 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:47.127877 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:32:47.127978 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:47.127940 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:32:47.127978 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:47.127967 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:32:48.069934 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:48.069898 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" event={"ID":"8a8f7a8d-d28e-45e5-be42-3137ba6e139b","Type":"ContainerStarted","Data":"820a63fe3f662f8a4396d806058f7771f4a32addf503d9b1ac074f2e566fb577"} Jul 06 19:32:48.070321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:48.069973 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" event={"ID":"8a8f7a8d-d28e-45e5-be42-3137ba6e139b","Type":"ContainerStarted","Data":"2d7b80980682d90ad34b99903ca4234e6dbbb527027e9c58f8238ebd2cfc989e"} Jul 06 19:32:48.610202 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:48.610153 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" podStartSLOduration=2.610138934 podStartE2EDuration="2.610138934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:48.609421377 +0000 UTC m=+578.251656203" watchObservedRunningTime="2026-07-06 19:32:48.610138934 +0000 UTC m=+578.252373759" Jul 06 19:32:48.987313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:48.987290 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:48.991893 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:48.991870 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:49.073501 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:49.073472 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:32:49.074293 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:32:49.074275 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:33:00.459182 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.459150 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-m8t6q"] Jul 06 19:33:00.461486 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.461464 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-m8t6q" Jul 06 19:33:00.464008 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.463985 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:33:00.464135 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.463984 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wlgsm\"" Jul 06 19:33:00.465060 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.465046 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:33:00.470331 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.470310 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-m8t6q"] Jul 06 19:33:00.549684 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.549631 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc-kube-api-access-r4kmt\") pod \"authorino-58cd955675-m8t6q\" (UID: \"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc\") " pod="kuadrant-system/authorino-58cd955675-m8t6q" Jul 06 19:33:00.650901 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.650849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc-kube-api-access-r4kmt\") pod \"authorino-58cd955675-m8t6q\" (UID: \"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc\") " pod="kuadrant-system/authorino-58cd955675-m8t6q" Jul 06 19:33:00.662647 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.662611 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc-kube-api-access-r4kmt\") pod \"authorino-58cd955675-m8t6q\" (UID: \"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc\") " pod="kuadrant-system/authorino-58cd955675-m8t6q" Jul 06 19:33:00.771236 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.771207 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-m8t6q" Jul 06 19:33:00.913349 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:00.913316 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-m8t6q"] Jul 06 19:33:00.916360 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:33:00.916335 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e19bbc1_5fa7_4f0d_a010_fd4d4a13fadc.slice/crio-f72df0f5cb1925a11b3098f70d9ecdc4edd9609f9ecb720b13f845a21b367466 WatchSource:0}: Error finding container f72df0f5cb1925a11b3098f70d9ecdc4edd9609f9ecb720b13f845a21b367466: Status 404 returned error can't find the container with id f72df0f5cb1925a11b3098f70d9ecdc4edd9609f9ecb720b13f845a21b367466 Jul 06 19:33:01.078676 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.078591 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-nqrkh"] Jul 06 19:33:01.081328 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.081312 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nqrkh" Jul 06 19:33:01.088460 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.088437 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nqrkh"] Jul 06 19:33:01.128004 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.127970 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-m8t6q" event={"ID":"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc","Type":"ContainerStarted","Data":"f72df0f5cb1925a11b3098f70d9ecdc4edd9609f9ecb720b13f845a21b367466"} Jul 06 19:33:01.156330 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.156296 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbcp\" (UniqueName: \"kubernetes.io/projected/412e62ee-c4d7-4a27-a794-f3d5ab050727-kube-api-access-fpbcp\") pod \"authorino-75d556bd77-nqrkh\" (UID: \"412e62ee-c4d7-4a27-a794-f3d5ab050727\") " pod="kuadrant-system/authorino-75d556bd77-nqrkh" Jul 06 19:33:01.257549 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.257513 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbcp\" (UniqueName: \"kubernetes.io/projected/412e62ee-c4d7-4a27-a794-f3d5ab050727-kube-api-access-fpbcp\") pod \"authorino-75d556bd77-nqrkh\" (UID: \"412e62ee-c4d7-4a27-a794-f3d5ab050727\") " pod="kuadrant-system/authorino-75d556bd77-nqrkh" Jul 06 19:33:01.266341 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.266304 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbcp\" (UniqueName: \"kubernetes.io/projected/412e62ee-c4d7-4a27-a794-f3d5ab050727-kube-api-access-fpbcp\") pod \"authorino-75d556bd77-nqrkh\" (UID: \"412e62ee-c4d7-4a27-a794-f3d5ab050727\") " pod="kuadrant-system/authorino-75d556bd77-nqrkh" Jul 06 19:33:01.390343 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.390254 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nqrkh" Jul 06 19:33:01.515942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:01.515881 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nqrkh"] Jul 06 19:33:01.517940 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:33:01.517912 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod412e62ee_c4d7_4a27_a794_f3d5ab050727.slice/crio-3ebd709228e2663e66893075e11f4fffa86c32157f20bbf1eb356930b8f313c6 WatchSource:0}: Error finding container 3ebd709228e2663e66893075e11f4fffa86c32157f20bbf1eb356930b8f313c6: Status 404 returned error can't find the container with id 3ebd709228e2663e66893075e11f4fffa86c32157f20bbf1eb356930b8f313c6 Jul 06 19:33:02.138853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:02.138787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nqrkh" event={"ID":"412e62ee-c4d7-4a27-a794-f3d5ab050727","Type":"ContainerStarted","Data":"3ebd709228e2663e66893075e11f4fffa86c32157f20bbf1eb356930b8f313c6"} Jul 06 19:33:05.156502 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:05.156450 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nqrkh" event={"ID":"412e62ee-c4d7-4a27-a794-f3d5ab050727","Type":"ContainerStarted","Data":"a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0"} Jul 06 19:33:05.157795 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:05.157772 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-m8t6q" event={"ID":"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc","Type":"ContainerStarted","Data":"dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe"} Jul 06 19:33:05.174957 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:05.174898 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-nqrkh" podStartSLOduration=1.333270601 podStartE2EDuration="4.174881879s" totalImagesPullingTime="2.841611278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:05.173566296 +0000 UTC m=+594.815801123" watchObservedRunningTime="2026-07-06 19:33:05.174881879 +0000 UTC m=+594.817116706" Jul 06 19:33:05.190129 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:05.190071 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-m8t6q" podStartSLOduration=1.7524692179999999 podStartE2EDuration="5.190056523s" totalImagesPullingTime="3.437587305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:05.189315108 +0000 UTC m=+594.831549936" watchObservedRunningTime="2026-07-06 19:33:05.190056523 +0000 UTC m=+594.832291348" Jul 06 19:33:05.220786 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:05.220739 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-m8t6q"] Jul 06 19:33:07.165291 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:07.165242 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-m8t6q" podUID="0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc" containerName="authorino" containerID="cri-o://dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe" gracePeriod=30 Jul 06 19:33:07.426250 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:07.426195 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-m8t6q" Jul 06 19:33:07.512687 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:07.512623 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc-kube-api-access-r4kmt\") pod \"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc\" (UID: \"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc\") " Jul 06 19:33:07.514858 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:07.514826 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc-kube-api-access-r4kmt" (OuterVolumeSpecName: "kube-api-access-r4kmt") pod "0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc" (UID: "0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc"). InnerVolumeSpecName "kube-api-access-r4kmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:07.613461 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:07.613407 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc-kube-api-access-r4kmt\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:33:08.170290 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.170252 2585 generic.go:356] "Generic (PLEG): container finished" podID="0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc" containerID="dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe" exitCode=0 Jul 06 19:33:08.170781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.170299 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-m8t6q" Jul 06 19:33:08.170781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.170334 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-m8t6q" event={"ID":"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc","Type":"ContainerDied","Data":"dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe"} Jul 06 19:33:08.170781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.170376 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-m8t6q" event={"ID":"0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc","Type":"ContainerDied","Data":"f72df0f5cb1925a11b3098f70d9ecdc4edd9609f9ecb720b13f845a21b367466"} Jul 06 19:33:08.170781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.170398 2585 scope.go:117] "RemoveContainer" containerID="dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe" Jul 06 19:33:08.178415 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.178397 2585 scope.go:117] "RemoveContainer" containerID="dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe" Jul 06 19:33:08.178693 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:33:08.178674 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe\": container with ID starting with dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe not found: ID does not exist" containerID="dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe" Jul 06 19:33:08.178757 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.178704 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe"} err="failed to get container status \"dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe\": rpc error: code = NotFound desc = could not find container \"dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe\": container with ID starting with dce091e326a22417beae7ec3b7f0e5ac4f27d2cb1bd264f66bf906631db77ebe not found: ID does not exist" Jul 06 19:33:08.192434 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.192408 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-m8t6q"] Jul 06 19:33:08.204216 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.204195 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-m8t6q"] Jul 06 19:33:08.872082 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:08.872046 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc" path="/var/lib/kubelet/pods/0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc/volumes" Jul 06 19:33:34.318560 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.318477 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ks7zw"] Jul 06 19:33:34.319149 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.318914 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc" containerName="authorino" Jul 06 19:33:34.319149 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.318934 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc" containerName="authorino" Jul 06 19:33:34.319149 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.319007 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e19bbc1-5fa7-4f0d-a010-fd4d4a13fadc" containerName="authorino" Jul 06 19:33:34.323948 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.323928 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" Jul 06 19:33:34.369410 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.369375 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ks7zw"] Jul 06 19:33:34.432819 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.432780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgp2\" (UniqueName: \"kubernetes.io/projected/10c461df-c1c2-49fd-9052-89cac17dccaf-kube-api-access-8wgp2\") pod \"authorino-5d4b67c446-ks7zw\" (UID: \"10c461df-c1c2-49fd-9052-89cac17dccaf\") " pod="kuadrant-system/authorino-5d4b67c446-ks7zw" Jul 06 19:33:34.533495 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.533459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgp2\" (UniqueName: \"kubernetes.io/projected/10c461df-c1c2-49fd-9052-89cac17dccaf-kube-api-access-8wgp2\") pod \"authorino-5d4b67c446-ks7zw\" (UID: \"10c461df-c1c2-49fd-9052-89cac17dccaf\") " pod="kuadrant-system/authorino-5d4b67c446-ks7zw" Jul 06 19:33:34.563675 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.563629 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgp2\" (UniqueName: \"kubernetes.io/projected/10c461df-c1c2-49fd-9052-89cac17dccaf-kube-api-access-8wgp2\") pod \"authorino-5d4b67c446-ks7zw\" (UID: \"10c461df-c1c2-49fd-9052-89cac17dccaf\") " pod="kuadrant-system/authorino-5d4b67c446-ks7zw" Jul 06 19:33:34.612883 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.612821 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ks7zw"] Jul 06 19:33:34.613044 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.613032 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" Jul 06 19:33:34.703889 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.703816 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d4f8c4f69-qzbnt"] Jul 06 19:33:34.707287 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.707267 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" Jul 06 19:33:34.735397 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.735375 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d4f8c4f69-qzbnt"] Jul 06 19:33:34.774570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.774548 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ks7zw"] Jul 06 19:33:34.777499 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:33:34.777472 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10c461df_c1c2_49fd_9052_89cac17dccaf.slice/crio-d30d16fc1d6e76f206f64c5f634a7da4695fb07545a1e35e58adfc22639cfd78 WatchSource:0}: Error finding container d30d16fc1d6e76f206f64c5f634a7da4695fb07545a1e35e58adfc22639cfd78: Status 404 returned error can't find the container with id d30d16fc1d6e76f206f64c5f634a7da4695fb07545a1e35e58adfc22639cfd78 Jul 06 19:33:34.836387 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.836359 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfv4\" (UniqueName: \"kubernetes.io/projected/b8b61e9a-af62-48ab-94db-caeb888d659a-kube-api-access-mxfv4\") pod \"authorino-d4f8c4f69-qzbnt\" (UID: \"b8b61e9a-af62-48ab-94db-caeb888d659a\") " pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" Jul 06 19:33:34.937668 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.937582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfv4\" (UniqueName: \"kubernetes.io/projected/b8b61e9a-af62-48ab-94db-caeb888d659a-kube-api-access-mxfv4\") pod \"authorino-d4f8c4f69-qzbnt\" (UID: \"b8b61e9a-af62-48ab-94db-caeb888d659a\") " pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" Jul 06 19:33:34.948914 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:34.948894 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfv4\" (UniqueName: \"kubernetes.io/projected/b8b61e9a-af62-48ab-94db-caeb888d659a-kube-api-access-mxfv4\") pod \"authorino-d4f8c4f69-qzbnt\" (UID: \"b8b61e9a-af62-48ab-94db-caeb888d659a\") " pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" Jul 06 19:33:35.019761 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.019738 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" Jul 06 19:33:35.079057 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.079030 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4f8c4f69-qzbnt"] Jul 06 19:33:35.108866 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.108841 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6945d6d485-98gf7"] Jul 06 19:33:35.111174 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.111152 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.113937 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.113911 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:33:35.120981 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.120958 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6945d6d485-98gf7"] Jul 06 19:33:35.146372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.146353 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4f8c4f69-qzbnt"] Jul 06 19:33:35.148779 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:33:35.148750 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8b61e9a_af62_48ab_94db_caeb888d659a.slice/crio-130ee5eb044004a3a3b73b7bd5fb64be4765f2ea9bdcea9b0622303372eaf61d WatchSource:0}: Error finding container 130ee5eb044004a3a3b73b7bd5fb64be4765f2ea9bdcea9b0622303372eaf61d: Status 404 returned error can't find the container with id 130ee5eb044004a3a3b73b7bd5fb64be4765f2ea9bdcea9b0622303372eaf61d Jul 06 19:33:35.240065 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.240029 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8874k\" (UniqueName: \"kubernetes.io/projected/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-kube-api-access-8874k\") pod \"authorino-6945d6d485-98gf7\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.240232 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.240094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-tls-cert\") pod \"authorino-6945d6d485-98gf7\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.285031 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.284997 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" event={"ID":"b8b61e9a-af62-48ab-94db-caeb888d659a","Type":"ContainerStarted","Data":"130ee5eb044004a3a3b73b7bd5fb64be4765f2ea9bdcea9b0622303372eaf61d"} Jul 06 19:33:35.286497 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.286473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" event={"ID":"10c461df-c1c2-49fd-9052-89cac17dccaf","Type":"ContainerStarted","Data":"350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d"} Jul 06 19:33:35.286497 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.286505 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" event={"ID":"10c461df-c1c2-49fd-9052-89cac17dccaf","Type":"ContainerStarted","Data":"d30d16fc1d6e76f206f64c5f634a7da4695fb07545a1e35e58adfc22639cfd78"} Jul 06 19:33:35.286670 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.286548 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" podUID="10c461df-c1c2-49fd-9052-89cac17dccaf" containerName="authorino" containerID="cri-o://350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d" gracePeriod=30 Jul 06 19:33:35.303332 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.303292 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" podStartSLOduration=0.960822335 podStartE2EDuration="1.303277702s" totalImagesPullingTime="342.455367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:35.301992536 +0000 UTC m=+624.944227361" watchObservedRunningTime="2026-07-06 19:33:35.303277702 +0000 UTC m=+624.945512529" Jul 06 19:33:35.341031 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.341009 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8874k\" (UniqueName: \"kubernetes.io/projected/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-kube-api-access-8874k\") pod \"authorino-6945d6d485-98gf7\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.341338 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.341069 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-tls-cert\") pod \"authorino-6945d6d485-98gf7\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.343300 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.343284 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-tls-cert\") pod \"authorino-6945d6d485-98gf7\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.351900 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.351880 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8874k\" (UniqueName: \"kubernetes.io/projected/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-kube-api-access-8874k\") pod \"authorino-6945d6d485-98gf7\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.423821 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.423787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:33:35.531196 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.531166 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" Jul 06 19:33:35.551069 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.551038 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6945d6d485-98gf7"] Jul 06 19:33:35.554684 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:33:35.554621 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5091e7c4_f643_4be0_ab86_45c70a8b2f9a.slice/crio-c27ea1dc8a74ed03f6efadc6aee4501d0eb41d6191fc46d1a4dbc70a973372bc WatchSource:0}: Error finding container c27ea1dc8a74ed03f6efadc6aee4501d0eb41d6191fc46d1a4dbc70a973372bc: Status 404 returned error can't find the container with id c27ea1dc8a74ed03f6efadc6aee4501d0eb41d6191fc46d1a4dbc70a973372bc Jul 06 19:33:35.643345 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.643319 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wgp2\" (UniqueName: \"kubernetes.io/projected/10c461df-c1c2-49fd-9052-89cac17dccaf-kube-api-access-8wgp2\") pod \"10c461df-c1c2-49fd-9052-89cac17dccaf\" (UID: \"10c461df-c1c2-49fd-9052-89cac17dccaf\") " Jul 06 19:33:35.645206 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.645186 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c461df-c1c2-49fd-9052-89cac17dccaf-kube-api-access-8wgp2" (OuterVolumeSpecName: "kube-api-access-8wgp2") pod "10c461df-c1c2-49fd-9052-89cac17dccaf" (UID: "10c461df-c1c2-49fd-9052-89cac17dccaf"). InnerVolumeSpecName "kube-api-access-8wgp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:35.744394 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:35.744367 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgp2\" (UniqueName: \"kubernetes.io/projected/10c461df-c1c2-49fd-9052-89cac17dccaf-kube-api-access-8wgp2\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:33:36.291726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.291687 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6945d6d485-98gf7" event={"ID":"5091e7c4-f643-4be0-ab86-45c70a8b2f9a","Type":"ContainerStarted","Data":"4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d"} Jul 06 19:33:36.291911 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.291737 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6945d6d485-98gf7" event={"ID":"5091e7c4-f643-4be0-ab86-45c70a8b2f9a","Type":"ContainerStarted","Data":"c27ea1dc8a74ed03f6efadc6aee4501d0eb41d6191fc46d1a4dbc70a973372bc"} Jul 06 19:33:36.293176 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.293149 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" event={"ID":"b8b61e9a-af62-48ab-94db-caeb888d659a","Type":"ContainerStarted","Data":"36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6"} Jul 06 19:33:36.293312 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.293197 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" podUID="b8b61e9a-af62-48ab-94db-caeb888d659a" containerName="authorino" containerID="cri-o://36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6" gracePeriod=30 Jul 06 19:33:36.294313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.294291 2585 generic.go:356] "Generic (PLEG): container finished" podID="10c461df-c1c2-49fd-9052-89cac17dccaf" containerID="350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d" exitCode=0 Jul 06 19:33:36.294413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.294330 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" Jul 06 19:33:36.294413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.294352 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" event={"ID":"10c461df-c1c2-49fd-9052-89cac17dccaf","Type":"ContainerDied","Data":"350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d"} Jul 06 19:33:36.294413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.294376 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ks7zw" event={"ID":"10c461df-c1c2-49fd-9052-89cac17dccaf","Type":"ContainerDied","Data":"d30d16fc1d6e76f206f64c5f634a7da4695fb07545a1e35e58adfc22639cfd78"} Jul 06 19:33:36.294413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.294397 2585 scope.go:117] "RemoveContainer" containerID="350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d" Jul 06 19:33:36.303309 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.303287 2585 scope.go:117] "RemoveContainer" containerID="350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d" Jul 06 19:33:36.303530 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:33:36.303511 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d\": container with ID starting with 350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d not found: ID does not exist" containerID="350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d" Jul 06 19:33:36.303582 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.303535 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d"} err="failed to get container status \"350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d\": rpc error: code = NotFound desc = could not find container \"350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d\": container with ID starting with 350841c35380b7ac09cf5652d0e1d93f4aea78fd1b8e1b2a9463b52f2000c92d not found: ID does not exist" Jul 06 19:33:36.309074 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.309039 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6945d6d485-98gf7" podStartSLOduration=0.999577566 podStartE2EDuration="1.309026994s" totalImagesPullingTime="309.449428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:36.307551487 +0000 UTC m=+625.949786315" watchObservedRunningTime="2026-07-06 19:33:36.309026994 +0000 UTC m=+625.951261819" Jul 06 19:33:36.325038 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.325013 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ks7zw"] Jul 06 19:33:36.329251 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.329232 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ks7zw"] Jul 06 19:33:36.331875 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.331856 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nqrkh"] Jul 06 19:33:36.332047 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.332025 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-nqrkh" podUID="412e62ee-c4d7-4a27-a794-f3d5ab050727" containerName="authorino" containerID="cri-o://a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0" gracePeriod=30 Jul 06 19:33:36.343231 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.343177 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" podStartSLOduration=1.8300998050000001 podStartE2EDuration="2.343162591s" totalImagesPullingTime="513.062786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:36.34295214 +0000 UTC m=+625.985186968" watchObservedRunningTime="2026-07-06 19:33:36.343162591 +0000 UTC m=+625.985397418" Jul 06 19:33:36.593610 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.593585 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" Jul 06 19:33:36.596241 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.596224 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nqrkh" Jul 06 19:33:36.753838 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.753807 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpbcp\" (UniqueName: \"kubernetes.io/projected/412e62ee-c4d7-4a27-a794-f3d5ab050727-kube-api-access-fpbcp\") pod \"412e62ee-c4d7-4a27-a794-f3d5ab050727\" (UID: \"412e62ee-c4d7-4a27-a794-f3d5ab050727\") " Jul 06 19:33:36.753977 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.753853 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxfv4\" (UniqueName: \"kubernetes.io/projected/b8b61e9a-af62-48ab-94db-caeb888d659a-kube-api-access-mxfv4\") pod \"b8b61e9a-af62-48ab-94db-caeb888d659a\" (UID: \"b8b61e9a-af62-48ab-94db-caeb888d659a\") " Jul 06 19:33:36.755816 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.755787 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b61e9a-af62-48ab-94db-caeb888d659a-kube-api-access-mxfv4" (OuterVolumeSpecName: "kube-api-access-mxfv4") pod "b8b61e9a-af62-48ab-94db-caeb888d659a" (UID: "b8b61e9a-af62-48ab-94db-caeb888d659a"). InnerVolumeSpecName "kube-api-access-mxfv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:36.755908 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.755818 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412e62ee-c4d7-4a27-a794-f3d5ab050727-kube-api-access-fpbcp" (OuterVolumeSpecName: "kube-api-access-fpbcp") pod "412e62ee-c4d7-4a27-a794-f3d5ab050727" (UID: "412e62ee-c4d7-4a27-a794-f3d5ab050727"). InnerVolumeSpecName "kube-api-access-fpbcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:33:36.854451 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.854394 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpbcp\" (UniqueName: \"kubernetes.io/projected/412e62ee-c4d7-4a27-a794-f3d5ab050727-kube-api-access-fpbcp\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:33:36.854451 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.854415 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfv4\" (UniqueName: \"kubernetes.io/projected/b8b61e9a-af62-48ab-94db-caeb888d659a-kube-api-access-mxfv4\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:33:36.872300 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:36.872274 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c461df-c1c2-49fd-9052-89cac17dccaf" path="/var/lib/kubelet/pods/10c461df-c1c2-49fd-9052-89cac17dccaf/volumes" Jul 06 19:33:37.300712 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.300675 2585 generic.go:356] "Generic (PLEG): container finished" podID="b8b61e9a-af62-48ab-94db-caeb888d659a" containerID="36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6" exitCode=0 Jul 06 19:33:37.300888 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.300725 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" Jul 06 19:33:37.300888 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.300746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" event={"ID":"b8b61e9a-af62-48ab-94db-caeb888d659a","Type":"ContainerDied","Data":"36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6"} Jul 06 19:33:37.300888 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.300780 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d4f8c4f69-qzbnt" event={"ID":"b8b61e9a-af62-48ab-94db-caeb888d659a","Type":"ContainerDied","Data":"130ee5eb044004a3a3b73b7bd5fb64be4765f2ea9bdcea9b0622303372eaf61d"} Jul 06 19:33:37.300888 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.300796 2585 scope.go:117] "RemoveContainer" containerID="36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6" Jul 06 19:33:37.302122 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.301927 2585 generic.go:356] "Generic (PLEG): container finished" podID="412e62ee-c4d7-4a27-a794-f3d5ab050727" containerID="a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0" exitCode=0 Jul 06 19:33:37.302122 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.301978 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nqrkh" Jul 06 19:33:37.302122 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.301998 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nqrkh" event={"ID":"412e62ee-c4d7-4a27-a794-f3d5ab050727","Type":"ContainerDied","Data":"a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0"} Jul 06 19:33:37.302122 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.302049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nqrkh" event={"ID":"412e62ee-c4d7-4a27-a794-f3d5ab050727","Type":"ContainerDied","Data":"3ebd709228e2663e66893075e11f4fffa86c32157f20bbf1eb356930b8f313c6"} Jul 06 19:33:37.308809 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.308789 2585 scope.go:117] "RemoveContainer" containerID="36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6" Jul 06 19:33:37.309054 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:33:37.309036 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6\": container with ID starting with 36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6 not found: ID does not exist" containerID="36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6" Jul 06 19:33:37.309106 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.309062 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6"} err="failed to get container status \"36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6\": rpc error: code = NotFound desc = could not find container \"36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6\": container with ID starting with 36aadeae13cdb571a74b3de43a8b1a0ad686fd1a4cd05c8344d5b96f5e4eecb6 not found: ID does not exist" Jul 06 19:33:37.309106 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.309077 2585 scope.go:117] "RemoveContainer" containerID="a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0" Jul 06 19:33:37.316296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.316274 2585 scope.go:117] "RemoveContainer" containerID="a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0" Jul 06 19:33:37.316439 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.316421 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4f8c4f69-qzbnt"] Jul 06 19:33:37.316528 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:33:37.316511 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0\": container with ID starting with a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0 not found: ID does not exist" containerID="a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0" Jul 06 19:33:37.316566 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.316535 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0"} err="failed to get container status \"a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0\": rpc error: code = NotFound desc = could not find container \"a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0\": container with ID starting with a83eb6c306156f88584942a558b3ef05586c6a3032ffeff75d6436da44fc81b0 not found: ID does not exist" Jul 06 19:33:37.319663 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.319635 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d4f8c4f69-qzbnt"] Jul 06 19:33:37.329646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.329623 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nqrkh"] Jul 06 19:33:37.332626 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:37.332607 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nqrkh"] Jul 06 19:33:38.872185 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:38.872143 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412e62ee-c4d7-4a27-a794-f3d5ab050727" path="/var/lib/kubelet/pods/412e62ee-c4d7-4a27-a794-f3d5ab050727/volumes" Jul 06 19:33:38.872707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:33:38.872567 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b61e9a-af62-48ab-94db-caeb888d659a" path="/var/lib/kubelet/pods/b8b61e9a-af62-48ab-94db-caeb888d659a/volumes" Jul 06 19:34:23.485125 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:23.485088 2585 generic.go:356] "Generic (PLEG): container finished" podID="8a8f7a8d-d28e-45e5-be42-3137ba6e139b" containerID="820a63fe3f662f8a4396d806058f7771f4a32addf503d9b1ac074f2e566fb577" exitCode=0 Jul 06 19:34:23.485514 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:23.485160 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" event={"ID":"8a8f7a8d-d28e-45e5-be42-3137ba6e139b","Type":"ContainerDied","Data":"820a63fe3f662f8a4396d806058f7771f4a32addf503d9b1ac074f2e566fb577"} Jul 06 19:34:23.485556 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:23.485526 2585 scope.go:117] "RemoveContainer" containerID="820a63fe3f662f8a4396d806058f7771f4a32addf503d9b1ac074f2e566fb577" Jul 06 19:34:23.486478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:23.486450 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:34:23.486537 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:23.486523 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:34:23.492104 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:23.491717 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:34:24.491456 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:24.491422 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" event={"ID":"8a8f7a8d-d28e-45e5-be42-3137ba6e139b","Type":"ContainerStarted","Data":"c64baa74b7f73c7b8d6f8b7d82f7825045866d8ba79ea103518f378086fef1e0"} Jul 06 19:34:24.987213 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:24.987182 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:34:24.987465 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:24.987442 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" start-of-body= Jul 06 19:34:24.987549 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:24.987515 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" podUID="8a8f7a8d-d28e-45e5-be42-3137ba6e139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" Jul 06 19:34:25.987845 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:25.987812 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" start-of-body= Jul 06 19:34:25.988237 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:25.987878 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" podUID="8a8f7a8d-d28e-45e5-be42-3137ba6e139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" Jul 06 19:34:26.987101 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:26.987069 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" start-of-body= Jul 06 19:34:26.987259 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:26.987115 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" podUID="8a8f7a8d-d28e-45e5-be42-3137ba6e139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" Jul 06 19:34:27.987756 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:27.987720 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" start-of-body= Jul 06 19:34:27.988140 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:27.987779 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" podUID="8a8f7a8d-d28e-45e5-be42-3137ba6e139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.20:15021/healthz/ready\": dial tcp 10.132.0.20:15021: connect: connection refused" Jul 06 19:34:28.990416 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:28.990385 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:34:28.990800 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:28.990737 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:34:28.991377 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:34:28.991359 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5dfc694f5c-cdq4j" Jul 06 19:35:22.281903 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.281819 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2"] Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282106 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412e62ee-c4d7-4a27-a794-f3d5ab050727" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282118 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="412e62ee-c4d7-4a27-a794-f3d5ab050727" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282128 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b61e9a-af62-48ab-94db-caeb888d659a" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282133 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b61e9a-af62-48ab-94db-caeb888d659a" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282145 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c461df-c1c2-49fd-9052-89cac17dccaf" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282151 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c461df-c1c2-49fd-9052-89cac17dccaf" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282203 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c461df-c1c2-49fd-9052-89cac17dccaf" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282211 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="412e62ee-c4d7-4a27-a794-f3d5ab050727" containerName="authorino" Jul 06 19:35:22.282492 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.282216 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b61e9a-af62-48ab-94db-caeb888d659a" containerName="authorino" Jul 06 19:35:22.285146 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.285127 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.289538 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.289510 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:35:22.289538 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.289537 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wxh4n\"" Jul 06 19:35:22.289761 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.289537 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 19:35:22.289761 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.289520 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:35:22.297412 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.297385 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2"] Jul 06 19:35:22.381699 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.381632 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.381699 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.381695 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.381921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.381733 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.381921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.381771 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7wg\" (UniqueName: \"kubernetes.io/projected/8998fd01-b071-4a81-a58f-c06e90ed0c21-kube-api-access-2x7wg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.381921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.381833 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8998fd01-b071-4a81-a58f-c06e90ed0c21-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.381921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.381889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.482983 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.482943 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8998fd01-b071-4a81-a58f-c06e90ed0c21-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483169 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483007 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483169 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483073 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483256 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483183 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483256 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483219 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483256 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483239 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7wg\" (UniqueName: \"kubernetes.io/projected/8998fd01-b071-4a81-a58f-c06e90ed0c21-kube-api-access-2x7wg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483480 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483459 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483595 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483576 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.483685 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.483609 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.485370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.485343 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8998fd01-b071-4a81-a58f-c06e90ed0c21-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.485532 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.485511 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8998fd01-b071-4a81-a58f-c06e90ed0c21-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.496458 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.496424 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7wg\" (UniqueName: \"kubernetes.io/projected/8998fd01-b071-4a81-a58f-c06e90ed0c21-kube-api-access-2x7wg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ctjd2\" (UID: \"8998fd01-b071-4a81-a58f-c06e90ed0c21\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.594643 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.594547 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:22.727721 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.727692 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2"] Jul 06 19:35:22.730147 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:35:22.730121 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8998fd01_b071_4a81_a58f_c06e90ed0c21.slice/crio-9e1c474a98ef3fa9c3213f6b621fa14667d0c509ac52e7041eb105d8596bf2e0 WatchSource:0}: Error finding container 9e1c474a98ef3fa9c3213f6b621fa14667d0c509ac52e7041eb105d8596bf2e0: Status 404 returned error can't find the container with id 9e1c474a98ef3fa9c3213f6b621fa14667d0c509ac52e7041eb105d8596bf2e0 Jul 06 19:35:22.731460 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.731443 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:35:22.740220 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:22.740187 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" event={"ID":"8998fd01-b071-4a81-a58f-c06e90ed0c21","Type":"ContainerStarted","Data":"9e1c474a98ef3fa9c3213f6b621fa14667d0c509ac52e7041eb105d8596bf2e0"} Jul 06 19:35:24.752266 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:24.752231 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" event={"ID":"8998fd01-b071-4a81-a58f-c06e90ed0c21","Type":"ContainerStarted","Data":"3f850160e8290c949da7f0faaf5470cd0316dfef001c26dc962d634b4635d0b4"} Jul 06 19:35:24.773353 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:24.773268 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" podStartSLOduration=1.655256947 podStartE2EDuration="2.773244581s" totalImagesPullingTime="1.117987634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:24.772837932 +0000 UTC m=+734.415072758" watchObservedRunningTime="2026-07-06 19:35:24.773244581 +0000 UTC m=+734.415479408" Jul 06 19:35:32.595026 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:32.594990 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:32.595481 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:32.595080 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:32.607789 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:32.607757 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:35:32.800862 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:35:32.800834 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ctjd2" Jul 06 19:36:25.093758 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.093725 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f5b64697b-dnjxw"] Jul 06 19:36:25.096876 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.096860 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.105169 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.105144 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5b64697b-dnjxw"] Jul 06 19:36:25.191773 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.191734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnvz\" (UniqueName: \"kubernetes.io/projected/b89b435a-1408-4ed2-8f88-bd2ad2f41caa-kube-api-access-zcnvz\") pod \"authorino-6f5b64697b-dnjxw\" (UID: \"b89b435a-1408-4ed2-8f88-bd2ad2f41caa\") " pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.191972 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.191802 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89b435a-1408-4ed2-8f88-bd2ad2f41caa-tls-cert\") pod \"authorino-6f5b64697b-dnjxw\" (UID: \"b89b435a-1408-4ed2-8f88-bd2ad2f41caa\") " pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.292738 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.292693 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnvz\" (UniqueName: \"kubernetes.io/projected/b89b435a-1408-4ed2-8f88-bd2ad2f41caa-kube-api-access-zcnvz\") pod \"authorino-6f5b64697b-dnjxw\" (UID: \"b89b435a-1408-4ed2-8f88-bd2ad2f41caa\") " pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.292914 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.292757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89b435a-1408-4ed2-8f88-bd2ad2f41caa-tls-cert\") pod \"authorino-6f5b64697b-dnjxw\" (UID: \"b89b435a-1408-4ed2-8f88-bd2ad2f41caa\") " pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.295250 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.295226 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89b435a-1408-4ed2-8f88-bd2ad2f41caa-tls-cert\") pod \"authorino-6f5b64697b-dnjxw\" (UID: \"b89b435a-1408-4ed2-8f88-bd2ad2f41caa\") " pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.301626 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.301604 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnvz\" (UniqueName: \"kubernetes.io/projected/b89b435a-1408-4ed2-8f88-bd2ad2f41caa-kube-api-access-zcnvz\") pod \"authorino-6f5b64697b-dnjxw\" (UID: \"b89b435a-1408-4ed2-8f88-bd2ad2f41caa\") " pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.405886 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.405808 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5b64697b-dnjxw" Jul 06 19:36:25.533184 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:25.533160 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5b64697b-dnjxw"] Jul 06 19:36:25.535715 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:36:25.535688 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89b435a_1408_4ed2_8f88_bd2ad2f41caa.slice/crio-a28c18ea8dac92629baffd30e44503e692e3769ed3532956e794d4c1e80b4a39 WatchSource:0}: Error finding container a28c18ea8dac92629baffd30e44503e692e3769ed3532956e794d4c1e80b4a39: Status 404 returned error can't find the container with id a28c18ea8dac92629baffd30e44503e692e3769ed3532956e794d4c1e80b4a39 Jul 06 19:36:26.000642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:26.000601 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5b64697b-dnjxw" event={"ID":"b89b435a-1408-4ed2-8f88-bd2ad2f41caa","Type":"ContainerStarted","Data":"a28c18ea8dac92629baffd30e44503e692e3769ed3532956e794d4c1e80b4a39"} Jul 06 19:36:27.008029 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.007995 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5b64697b-dnjxw" event={"ID":"b89b435a-1408-4ed2-8f88-bd2ad2f41caa","Type":"ContainerStarted","Data":"8869a4a3dc027e1f85a8a012d33327cea9f88fc3e8ef43b0eabe672f7eed3f06"} Jul 06 19:36:27.044675 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.044592 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f5b64697b-dnjxw" podStartSLOduration=1.564168427 podStartE2EDuration="2.044572347s" totalImagesPullingTime="480.40392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:27.044077809 +0000 UTC m=+796.686312634" watchObservedRunningTime="2026-07-06 19:36:27.044572347 +0000 UTC m=+796.686807176" Jul 06 19:36:27.111110 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.111068 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6945d6d485-98gf7"] Jul 06 19:36:27.111306 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.111274 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6945d6d485-98gf7" podUID="5091e7c4-f643-4be0-ab86-45c70a8b2f9a" containerName="authorino" containerID="cri-o://4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d" gracePeriod=30 Jul 06 19:36:27.358453 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.358429 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:36:27.411457 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.411423 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-tls-cert\") pod \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " Jul 06 19:36:27.411624 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.411553 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8874k\" (UniqueName: \"kubernetes.io/projected/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-kube-api-access-8874k\") pod \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\" (UID: \"5091e7c4-f643-4be0-ab86-45c70a8b2f9a\") " Jul 06 19:36:27.413647 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.413617 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-kube-api-access-8874k" (OuterVolumeSpecName: "kube-api-access-8874k") pod "5091e7c4-f643-4be0-ab86-45c70a8b2f9a" (UID: "5091e7c4-f643-4be0-ab86-45c70a8b2f9a"). InnerVolumeSpecName "kube-api-access-8874k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:36:27.420809 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.420776 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5091e7c4-f643-4be0-ab86-45c70a8b2f9a" (UID: "5091e7c4-f643-4be0-ab86-45c70a8b2f9a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:36:27.512401 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.512361 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8874k\" (UniqueName: \"kubernetes.io/projected/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-kube-api-access-8874k\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:36:27.512401 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:27.512394 2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5091e7c4-f643-4be0-ab86-45c70a8b2f9a-tls-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:36:28.014157 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.014125 2585 generic.go:356] "Generic (PLEG): container finished" podID="5091e7c4-f643-4be0-ab86-45c70a8b2f9a" containerID="4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d" exitCode=0 Jul 06 19:36:28.014608 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.014166 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6945d6d485-98gf7" Jul 06 19:36:28.014608 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.014224 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6945d6d485-98gf7" event={"ID":"5091e7c4-f643-4be0-ab86-45c70a8b2f9a","Type":"ContainerDied","Data":"4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d"} Jul 06 19:36:28.014608 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.014268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6945d6d485-98gf7" event={"ID":"5091e7c4-f643-4be0-ab86-45c70a8b2f9a","Type":"ContainerDied","Data":"c27ea1dc8a74ed03f6efadc6aee4501d0eb41d6191fc46d1a4dbc70a973372bc"} Jul 06 19:36:28.014608 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.014294 2585 scope.go:117] "RemoveContainer" containerID="4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d" Jul 06 19:36:28.022522 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.022506 2585 scope.go:117] "RemoveContainer" containerID="4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d" Jul 06 19:36:28.022794 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:36:28.022775 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d\": container with ID starting with 4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d not found: ID does not exist" containerID="4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d" Jul 06 19:36:28.022883 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.022801 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d"} err="failed to get container status \"4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d\": rpc error: code = NotFound desc = could not find container \"4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d\": container with ID starting with 4cdf81550553355609e88322108c8cab1ce17ac499ef14d79c10826af2cb4b5d not found: ID does not exist" Jul 06 19:36:28.038778 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.038755 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6945d6d485-98gf7"] Jul 06 19:36:28.043953 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.043933 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6945d6d485-98gf7"] Jul 06 19:36:28.872331 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:36:28.872300 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5091e7c4-f643-4be0-ab86-45c70a8b2f9a" path="/var/lib/kubelet/pods/5091e7c4-f643-4be0-ab86-45c70a8b2f9a/volumes" Jul 06 19:37:27.258917 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:37:27.258882 2585 generic.go:356] "Generic (PLEG): container finished" podID="d6c1bf2d-7302-4f07-99bb-db4a88b6351b" containerID="ca85f8744c3b06a289dfe104d9e1b9c061bf0969b7288a2330c5a61d62748256" exitCode=1 Jul 06 19:37:27.259329 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:37:27.258951 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" event={"ID":"d6c1bf2d-7302-4f07-99bb-db4a88b6351b","Type":"ContainerDied","Data":"ca85f8744c3b06a289dfe104d9e1b9c061bf0969b7288a2330c5a61d62748256"} Jul 06 19:37:27.259329 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:37:27.259253 2585 scope.go:117] "RemoveContainer" containerID="ca85f8744c3b06a289dfe104d9e1b9c061bf0969b7288a2330c5a61d62748256" Jul 06 19:37:28.264722 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:37:28.264681 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" event={"ID":"d6c1bf2d-7302-4f07-99bb-db4a88b6351b","Type":"ContainerStarted","Data":"12cfca3fdc917157d93bb6d79b90acf21ed7e4153aca7d27fdf7d0b49749ef9c"} Jul 06 19:37:28.265092 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:37:28.264933 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:37:39.271510 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:37:39.271475 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mrp5j" Jul 06 19:45:00.160684 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.160638 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-c4rh9"] Jul 06 19:45:00.161286 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.161066 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5091e7c4-f643-4be0-ab86-45c70a8b2f9a" containerName="authorino" Jul 06 19:45:00.161286 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.161085 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5091e7c4-f643-4be0-ab86-45c70a8b2f9a" containerName="authorino" Jul 06 19:45:00.161286 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.161182 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5091e7c4-f643-4be0-ab86-45c70a8b2f9a" containerName="authorino" Jul 06 19:45:00.164259 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.164239 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" Jul 06 19:45:00.167375 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.167355 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jplkb\"" Jul 06 19:45:00.174796 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.174762 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-c4rh9"] Jul 06 19:45:00.278048 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.278017 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cxvm\" (UniqueName: \"kubernetes.io/projected/9e5ad88d-e037-4ad8-90c2-db6f9760a81c-kube-api-access-9cxvm\") pod \"maas-api-key-cleanup-29722785-c4rh9\" (UID: \"9e5ad88d-e037-4ad8-90c2-db6f9760a81c\") " pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" Jul 06 19:45:00.378750 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.378710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cxvm\" (UniqueName: \"kubernetes.io/projected/9e5ad88d-e037-4ad8-90c2-db6f9760a81c-kube-api-access-9cxvm\") pod \"maas-api-key-cleanup-29722785-c4rh9\" (UID: \"9e5ad88d-e037-4ad8-90c2-db6f9760a81c\") " pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" Jul 06 19:45:00.388289 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.388261 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cxvm\" (UniqueName: \"kubernetes.io/projected/9e5ad88d-e037-4ad8-90c2-db6f9760a81c-kube-api-access-9cxvm\") pod \"maas-api-key-cleanup-29722785-c4rh9\" (UID: \"9e5ad88d-e037-4ad8-90c2-db6f9760a81c\") " pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" Jul 06 19:45:00.474491 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.474408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" Jul 06 19:45:00.608780 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.608620 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-c4rh9"] Jul 06 19:45:00.611380 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:45:00.611354 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5ad88d_e037_4ad8_90c2_db6f9760a81c.slice/crio-ec6c61f74976cb027b84914814e6fb25823d8be246690d8dbd050fe08c0645d5 WatchSource:0}: Error finding container ec6c61f74976cb027b84914814e6fb25823d8be246690d8dbd050fe08c0645d5: Status 404 returned error can't find the container with id ec6c61f74976cb027b84914814e6fb25823d8be246690d8dbd050fe08c0645d5 Jul 06 19:45:00.613267 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:00.613249 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:45:01.172259 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:01.172216 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" event={"ID":"9e5ad88d-e037-4ad8-90c2-db6f9760a81c","Type":"ContainerStarted","Data":"ec6c61f74976cb027b84914814e6fb25823d8be246690d8dbd050fe08c0645d5"} Jul 06 19:45:03.186736 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:03.186636 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" event={"ID":"9e5ad88d-e037-4ad8-90c2-db6f9760a81c","Type":"ContainerStarted","Data":"e89ce7231d563986b1c9605ee5debda22d2bf6b31529785dfe73d2b9cc9f46fb"} Jul 06 19:45:03.204747 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:03.204699 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" podStartSLOduration=1.144240587 podStartE2EDuration="3.204685571s" totalImagesPullingTime="2.060444984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:03.203483878 +0000 UTC m=+1312.845718712" watchObservedRunningTime="2026-07-06 19:45:03.204685571 +0000 UTC m=+1312.846920394" Jul 06 19:45:04.191457 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:04.191419 2585 generic.go:356] "Generic (PLEG): container finished" podID="9e5ad88d-e037-4ad8-90c2-db6f9760a81c" containerID="e89ce7231d563986b1c9605ee5debda22d2bf6b31529785dfe73d2b9cc9f46fb" exitCode=0 Jul 06 19:45:04.191849 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:04.191507 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" event={"ID":"9e5ad88d-e037-4ad8-90c2-db6f9760a81c","Type":"ContainerDied","Data":"e89ce7231d563986b1c9605ee5debda22d2bf6b31529785dfe73d2b9cc9f46fb"} Jul 06 19:45:05.320551 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:05.320527 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" Jul 06 19:45:05.420612 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:05.420581 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cxvm\" (UniqueName: \"kubernetes.io/projected/9e5ad88d-e037-4ad8-90c2-db6f9760a81c-kube-api-access-9cxvm\") pod \"9e5ad88d-e037-4ad8-90c2-db6f9760a81c\" (UID: \"9e5ad88d-e037-4ad8-90c2-db6f9760a81c\") " Jul 06 19:45:05.422795 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:05.422764 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5ad88d-e037-4ad8-90c2-db6f9760a81c-kube-api-access-9cxvm" (OuterVolumeSpecName: "kube-api-access-9cxvm") pod "9e5ad88d-e037-4ad8-90c2-db6f9760a81c" (UID: "9e5ad88d-e037-4ad8-90c2-db6f9760a81c"). InnerVolumeSpecName "kube-api-access-9cxvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:05.521192 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:05.521158 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cxvm\" (UniqueName: \"kubernetes.io/projected/9e5ad88d-e037-4ad8-90c2-db6f9760a81c-kube-api-access-9cxvm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:45:06.201091 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:06.201055 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" Jul 06 19:45:06.201271 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:06.201095 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-c4rh9" event={"ID":"9e5ad88d-e037-4ad8-90c2-db6f9760a81c","Type":"ContainerDied","Data":"ec6c61f74976cb027b84914814e6fb25823d8be246690d8dbd050fe08c0645d5"} Jul 06 19:45:06.201271 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:45:06.201148 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6c61f74976cb027b84914814e6fb25823d8be246690d8dbd050fe08c0645d5" Jul 06 19:50:40.412282 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.412249 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9"] Jul 06 19:50:40.414322 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.412540 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5ad88d-e037-4ad8-90c2-db6f9760a81c" containerName="cleanup" Jul 06 19:50:40.414322 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.412551 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5ad88d-e037-4ad8-90c2-db6f9760a81c" containerName="cleanup" Jul 06 19:50:40.414322 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.412602 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5ad88d-e037-4ad8-90c2-db6f9760a81c" containerName="cleanup" Jul 06 19:50:40.415268 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.415252 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.419482 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.419462 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b1cbd1f8-openshift-default-dockercfg-ch2jp\"" Jul 06 19:50:40.445517 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.445486 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9"] Jul 06 19:50:40.451073 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451046 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-credential-socket\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451076 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhl8\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-kube-api-access-hlhl8\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451109 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-token\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451180 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-socket\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-data\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451504 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451226 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-podinfo\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451504 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451262 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istiod-ca-cert\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451504 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451280 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-certs\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.451504 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.451333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-envoy\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551771 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551737 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-certs\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551970 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-envoy\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551970 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-credential-socket\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551970 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhl8\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-kube-api-access-hlhl8\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551970 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-token\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551970 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-socket\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551970 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-data\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.551970 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551939 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-podinfo\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.552332 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.551987 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istiod-ca-cert\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.552332 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.552195 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-certs\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.552332 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.552260 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-credential-socket\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.552479 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.552425 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-data\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.552479 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.552464 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-socket\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.552642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.552625 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istiod-ca-cert\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.554251 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.554231 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-envoy\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.554434 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.554417 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-podinfo\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.562071 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.562046 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhl8\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-kube-api-access-hlhl8\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.566817 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.566792 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-token\") pod \"e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.725153 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.725038 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:40.860743 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.860719 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9"] Jul 06 19:50:40.862512 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:50:40.862479 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfaa0e0_9099_48b3_b539_85398c6a6d07.slice/crio-8e1aa266a02c318f230929d877be24bc3305604610ceed69f8398735ba79ae5a WatchSource:0}: Error finding container 8e1aa266a02c318f230929d877be24bc3305604610ceed69f8398735ba79ae5a: Status 404 returned error can't find the container with id 8e1aa266a02c318f230929d877be24bc3305604610ceed69f8398735ba79ae5a Jul 06 19:50:40.864401 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.864372 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:50:40.864472 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.864437 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:50:40.864508 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:40.864487 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:50:41.623506 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:41.623470 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" event={"ID":"7bfaa0e0-9099-48b3-b539-85398c6a6d07","Type":"ContainerStarted","Data":"e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b"} Jul 06 19:50:41.623996 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:41.623521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" event={"ID":"7bfaa0e0-9099-48b3-b539-85398c6a6d07","Type":"ContainerStarted","Data":"8e1aa266a02c318f230929d877be24bc3305604610ceed69f8398735ba79ae5a"} Jul 06 19:50:41.649973 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:41.649927 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" podStartSLOduration=1.64991136 podStartE2EDuration="1.64991136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:50:41.648134499 +0000 UTC m=+1651.290369326" watchObservedRunningTime="2026-07-06 19:50:41.64991136 +0000 UTC m=+1651.292146187" Jul 06 19:50:41.725618 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:41.725588 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:41.730099 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:41.730076 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:42.628399 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:42.628364 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:42.629231 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:42.629213 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:46.856149 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:46.856105 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9"] Jul 06 19:50:46.856799 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:46.856349 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" podUID="7bfaa0e0-9099-48b3-b539-85398c6a6d07" containerName="istio-proxy" containerID="cri-o://e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b" gracePeriod=30 Jul 06 19:50:52.504983 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.504949 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:52.557505 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557419 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-certs\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557505 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557457 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-socket\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557505 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557484 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlhl8\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-kube-api-access-hlhl8\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557514 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istiod-ca-cert\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557554 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-podinfo\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557570 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-envoy\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557614 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-credential-socket\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557646 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-data\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557727 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-token\") pod \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\" (UID: \"7bfaa0e0-9099-48b3-b539-85398c6a6d07\") " Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557762 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:52.557884 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557775 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:52.558304 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557976 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.558304 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557994 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.558304 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.557995 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:52.558304 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.558261 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:50:52.558592 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.558567 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:52.560138 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.560112 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-kube-api-access-hlhl8" (OuterVolumeSpecName: "kube-api-access-hlhl8") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "kube-api-access-hlhl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:52.560138 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.560125 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:52.560311 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.560289 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:52.560311 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.560300 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bfaa0e0-9099-48b3-b539-85398c6a6d07" (UID: "7bfaa0e0-9099-48b3-b539-85398c6a6d07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:50:52.658799 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.658767 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.658799 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.658790 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.658799 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.658800 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhl8\" (UniqueName: \"kubernetes.io/projected/7bfaa0e0-9099-48b3-b539-85398c6a6d07-kube-api-access-hlhl8\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.658799 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.658809 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.659048 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.658817 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.659048 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.658825 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.659048 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.658833 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfaa0e0-9099-48b3-b539-85398c6a6d07-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:50:52.677575 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.677544 2585 generic.go:356] "Generic (PLEG): container finished" podID="7bfaa0e0-9099-48b3-b539-85398c6a6d07" containerID="e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b" exitCode=0 Jul 06 19:50:52.677729 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.677590 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" Jul 06 19:50:52.677729 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.677636 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" event={"ID":"7bfaa0e0-9099-48b3-b539-85398c6a6d07","Type":"ContainerDied","Data":"e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b"} Jul 06 19:50:52.677729 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.677701 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9" event={"ID":"7bfaa0e0-9099-48b3-b539-85398c6a6d07","Type":"ContainerDied","Data":"8e1aa266a02c318f230929d877be24bc3305604610ceed69f8398735ba79ae5a"} Jul 06 19:50:52.677729 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.677727 2585 scope.go:117] "RemoveContainer" containerID="e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b" Jul 06 19:50:52.686776 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.686760 2585 scope.go:117] "RemoveContainer" containerID="e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b" Jul 06 19:50:52.687025 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:50:52.687008 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b\": container with ID starting with e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b not found: ID does not exist" containerID="e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b" Jul 06 19:50:52.687079 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.687032 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b"} err="failed to get container status \"e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b\": rpc error: code = NotFound desc = could not find container \"e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b\": container with ID starting with e786ec79ef9f95ea7509b8433b168490789f16f8f965bfadd320cd244d815c3b not found: ID does not exist" Jul 06 19:50:52.710124 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.710098 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9"] Jul 06 19:50:52.718083 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.718057 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b1cbd1f8-openshift-default-77bff567bf-bfvd9"] Jul 06 19:50:52.875277 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:52.875171 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bfaa0e0-9099-48b3-b539-85398c6a6d07" path="/var/lib/kubelet/pods/7bfaa0e0-9099-48b3-b539-85398c6a6d07/volumes" Jul 06 19:50:54.097455 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.097418 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt"] Jul 06 19:50:54.097860 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.097722 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bfaa0e0-9099-48b3-b539-85398c6a6d07" containerName="istio-proxy" Jul 06 19:50:54.097860 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.097733 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bfaa0e0-9099-48b3-b539-85398c6a6d07" containerName="istio-proxy" Jul 06 19:50:54.097860 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.097789 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bfaa0e0-9099-48b3-b539-85398c6a6d07" containerName="istio-proxy" Jul 06 19:50:54.100481 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.100465 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.103462 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.103442 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-174709cd-openshift-default-dockercfg-v64cl\"" Jul 06 19:50:54.117024 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.116975 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt"] Jul 06 19:50:54.170891 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.170859 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-data\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.170891 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.170889 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-token\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.171069 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.170909 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-envoy\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.171069 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.170937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-podinfo\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.171069 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.171011 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-credential-socket\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.171069 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.171044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkw9d\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-kube-api-access-rkw9d\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.171196 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.171074 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-certs\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.171196 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.171114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-socket\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.171196 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.171133 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067ec16e-37eb-48ba-b10e-4c361838ae97-istiod-ca-cert\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271543 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-credential-socket\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271543 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271545 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkw9d\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-kube-api-access-rkw9d\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271813 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-certs\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271813 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-socket\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271813 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067ec16e-37eb-48ba-b10e-4c361838ae97-istiod-ca-cert\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271813 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271737 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-data\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271813 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-token\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.271813 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271786 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-envoy\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.272117 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-podinfo\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.272117 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.272001 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-certs\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.272117 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.271998 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-credential-socket\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.272259 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.272246 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-data\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.272321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.272276 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-socket\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.272478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.272458 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067ec16e-37eb-48ba-b10e-4c361838ae97-istiod-ca-cert\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.274233 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.274210 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-envoy\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.274387 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.274368 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-podinfo\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.281187 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.281165 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-token\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.281418 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.281398 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkw9d\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-kube-api-access-rkw9d\") pod \"e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.409965 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.409878 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:54.551458 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.551435 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt"] Jul 06 19:50:54.553171 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:50:54.553144 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067ec16e_37eb_48ba_b10e_4c361838ae97.slice/crio-3fae602efb08ca10cb36625484c17f4bfa9cf48ab78e2baee2a3f2014cc20787 WatchSource:0}: Error finding container 3fae602efb08ca10cb36625484c17f4bfa9cf48ab78e2baee2a3f2014cc20787: Status 404 returned error can't find the container with id 3fae602efb08ca10cb36625484c17f4bfa9cf48ab78e2baee2a3f2014cc20787 Jul 06 19:50:54.555300 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.555271 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:50:54.555381 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.555334 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:50:54.555381 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.555364 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:50:54.689041 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.688951 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" event={"ID":"067ec16e-37eb-48ba-b10e-4c361838ae97","Type":"ContainerStarted","Data":"0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c"} Jul 06 19:50:54.689041 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.689014 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" event={"ID":"067ec16e-37eb-48ba-b10e-4c361838ae97","Type":"ContainerStarted","Data":"3fae602efb08ca10cb36625484c17f4bfa9cf48ab78e2baee2a3f2014cc20787"} Jul 06 19:50:54.715455 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:54.715399 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" podStartSLOduration=0.715381406 podStartE2EDuration="715.381406ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:50:54.71422247 +0000 UTC m=+1664.356457297" watchObservedRunningTime="2026-07-06 19:50:54.715381406 +0000 UTC m=+1664.357616231" Jul 06 19:50:55.410418 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:55.410376 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:55.415567 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:55.415542 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:55.693804 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:55.693710 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:50:55.694555 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:50:55.694535 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:51:06.476462 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:06.476424 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt"] Jul 06 19:51:06.477106 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:06.476648 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" podUID="067ec16e-37eb-48ba-b10e-4c361838ae97" containerName="istio-proxy" containerID="cri-o://0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c" gracePeriod=30 Jul 06 19:51:08.335574 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.335552 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:51:08.504200 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504159 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-credential-socket\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504216 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067ec16e-37eb-48ba-b10e-4c361838ae97-istiod-ca-cert\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504247 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-envoy\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504275 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-podinfo\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-certs\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504372 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504319 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-socket\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504451 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkw9d\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-kube-api-access-rkw9d\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504475 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:08.504708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504498 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-token\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504559 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-data\") pod \"067ec16e-37eb-48ba-b10e-4c361838ae97\" (UID: \"067ec16e-37eb-48ba-b10e-4c361838ae97\") " Jul 06 19:51:08.504708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504598 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:08.504708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504614 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:08.504708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504682 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067ec16e-37eb-48ba-b10e-4c361838ae97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:08.505018 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504892 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.505018 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504912 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.505018 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504928 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.505018 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.504942 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/067ec16e-37eb-48ba-b10e-4c361838ae97-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.505413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.505388 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-data" (OuterVolumeSpecName: "istio-data") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:08.506761 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.506728 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:08.506895 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.506791 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-token" (OuterVolumeSpecName: "istio-token") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:08.507065 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.506966 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-kube-api-access-rkw9d" (OuterVolumeSpecName: "kube-api-access-rkw9d") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "kube-api-access-rkw9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:08.507065 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.507051 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "067ec16e-37eb-48ba-b10e-4c361838ae97" (UID: "067ec16e-37eb-48ba-b10e-4c361838ae97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:51:08.605820 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.605789 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.605820 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.605814 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.605985 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.605826 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkw9d\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-kube-api-access-rkw9d\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.605985 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.605839 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.605985 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.605850 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/067ec16e-37eb-48ba-b10e-4c361838ae97-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:08.758096 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.758008 2585 generic.go:356] "Generic (PLEG): container finished" podID="067ec16e-37eb-48ba-b10e-4c361838ae97" containerID="0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c" exitCode=0 Jul 06 19:51:08.758096 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.758058 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" Jul 06 19:51:08.758305 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.758089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" event={"ID":"067ec16e-37eb-48ba-b10e-4c361838ae97","Type":"ContainerDied","Data":"0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c"} Jul 06 19:51:08.758305 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.758145 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt" event={"ID":"067ec16e-37eb-48ba-b10e-4c361838ae97","Type":"ContainerDied","Data":"3fae602efb08ca10cb36625484c17f4bfa9cf48ab78e2baee2a3f2014cc20787"} Jul 06 19:51:08.758305 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.758166 2585 scope.go:117] "RemoveContainer" containerID="0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c" Jul 06 19:51:08.766568 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.766554 2585 scope.go:117] "RemoveContainer" containerID="0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c" Jul 06 19:51:08.766819 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:51:08.766801 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c\": container with ID starting with 0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c not found: ID does not exist" containerID="0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c" Jul 06 19:51:08.766887 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.766825 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c"} err="failed to get container status \"0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c\": rpc error: code = NotFound desc = could not find container \"0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c\": container with ID starting with 0307200c0d2a010a0c829e54823473e1e8579b69cc6afe5949fa82c817db261c not found: ID does not exist" Jul 06 19:51:08.783988 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.783966 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt"] Jul 06 19:51:08.806705 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.806679 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-174709cd-openshift-default-597b78c7b5-2vljt"] Jul 06 19:51:08.872199 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:08.872163 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067ec16e-37eb-48ba-b10e-4c361838ae97" path="/var/lib/kubelet/pods/067ec16e-37eb-48ba-b10e-4c361838ae97/volumes" Jul 06 19:51:15.319362 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.319329 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn"] Jul 06 19:51:15.319733 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.319624 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067ec16e-37eb-48ba-b10e-4c361838ae97" containerName="istio-proxy" Jul 06 19:51:15.319733 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.319634 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="067ec16e-37eb-48ba-b10e-4c361838ae97" containerName="istio-proxy" Jul 06 19:51:15.319733 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.319702 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="067ec16e-37eb-48ba-b10e-4c361838ae97" containerName="istio-proxy" Jul 06 19:51:15.321475 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.321458 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.328044 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.328015 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-22b142fd-openshift-default-dockercfg-np84q\"" Jul 06 19:51:15.339698 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.339672 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn"] Jul 06 19:51:15.362382 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362358 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-envoy\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362485 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362391 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-token\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362485 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362422 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-data\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-credential-socket\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362547 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247556b-2934-4525-8ec4-fa57d77fa02e-istiod-ca-cert\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362614 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-socket\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362789 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362641 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-podinfo\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362789 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362698 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-certs\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.362789 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.362726 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfg8z\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-kube-api-access-dfg8z\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463213 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463184 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-credential-socket\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463396 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463221 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247556b-2934-4525-8ec4-fa57d77fa02e-istiod-ca-cert\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463396 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463246 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-socket\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463396 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463263 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-podinfo\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463396 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463284 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-certs\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463396 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463299 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfg8z\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-kube-api-access-dfg8z\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463425 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-envoy\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463469 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-token\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463642 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-data\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463823 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463649 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-socket\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463823 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463800 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-credential-socket\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463919 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463903 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-data\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.463974 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.463955 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-certs\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.464148 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.464128 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247556b-2934-4525-8ec4-fa57d77fa02e-istiod-ca-cert\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.465746 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.465716 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-envoy\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.465881 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.465865 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-podinfo\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.480145 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.480118 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-token\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.480260 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.480244 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfg8z\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-kube-api-access-dfg8z\") pod \"e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.631707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.631620 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:15.768740 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.768712 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn"] Jul 06 19:51:15.770541 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:51:15.770518 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda247556b_2934_4525_8ec4_fa57d77fa02e.slice/crio-7ab4920e5a831fd1290535dac6c4cd6d3eef428cc82cf9af9f554ec96ddca091 WatchSource:0}: Error finding container 7ab4920e5a831fd1290535dac6c4cd6d3eef428cc82cf9af9f554ec96ddca091: Status 404 returned error can't find the container with id 7ab4920e5a831fd1290535dac6c4cd6d3eef428cc82cf9af9f554ec96ddca091 Jul 06 19:51:15.772604 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.772565 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:15.772697 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.772631 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:15.772762 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.772699 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:15.789351 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:15.789320 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" event={"ID":"a247556b-2934-4525-8ec4-fa57d77fa02e","Type":"ContainerStarted","Data":"7ab4920e5a831fd1290535dac6c4cd6d3eef428cc82cf9af9f554ec96ddca091"} Jul 06 19:51:16.795716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:16.795681 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" event={"ID":"a247556b-2934-4525-8ec4-fa57d77fa02e","Type":"ContainerStarted","Data":"cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f"} Jul 06 19:51:16.830456 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:16.830394 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" podStartSLOduration=1.830375879 podStartE2EDuration="1.830375879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:16.828791512 +0000 UTC m=+1686.471026339" watchObservedRunningTime="2026-07-06 19:51:16.830375879 +0000 UTC m=+1686.472610706" Jul 06 19:51:17.632065 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:17.632030 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:17.636342 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:17.636318 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:17.800754 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:17.800725 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:17.801494 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:17.801476 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:21.151872 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:21.151831 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn"] Jul 06 19:51:21.152793 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:21.152736 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" podUID="a247556b-2934-4525-8ec4-fa57d77fa02e" containerName="istio-proxy" containerID="cri-o://cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f" gracePeriod=30 Jul 06 19:51:27.496193 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.496133 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:27.572001 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.571967 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-data\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572171 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572024 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfg8z\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-kube-api-access-dfg8z\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572171 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572064 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-socket\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572171 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572085 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-token\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572171 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572114 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-podinfo\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572171 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572136 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-certs\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572434 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572188 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-credential-socket\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572434 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572225 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-envoy\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572434 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572269 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247556b-2934-4525-8ec4-fa57d77fa02e-istiod-ca-cert\") pod \"a247556b-2934-4525-8ec4-fa57d77fa02e\" (UID: \"a247556b-2934-4525-8ec4-fa57d77fa02e\") " Jul 06 19:51:27.572592 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572480 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:27.572592 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572536 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:27.573296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.573215 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-data" (OuterVolumeSpecName: "istio-data") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:27.573296 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.573252 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a247556b-2934-4525-8ec4-fa57d77fa02e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:27.573506 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.572594 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:27.575385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.575359 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-token" (OuterVolumeSpecName: "istio-token") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:27.579807 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.579780 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:51:27.579807 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.579793 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:27.579942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.579793 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-kube-api-access-dfg8z" (OuterVolumeSpecName: "kube-api-access-dfg8z") pod "a247556b-2934-4525-8ec4-fa57d77fa02e" (UID: "a247556b-2934-4525-8ec4-fa57d77fa02e"). InnerVolumeSpecName "kube-api-access-dfg8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:27.673641 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673606 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673641 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673636 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a247556b-2934-4525-8ec4-fa57d77fa02e-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673641 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673646 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673896 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673675 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfg8z\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-kube-api-access-dfg8z\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673896 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673685 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673896 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673694 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673896 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673701 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a247556b-2934-4525-8ec4-fa57d77fa02e-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673896 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673709 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.673896 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.673717 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a247556b-2934-4525-8ec4-fa57d77fa02e-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:51:27.844121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.844035 2585 generic.go:356] "Generic (PLEG): container finished" podID="a247556b-2934-4525-8ec4-fa57d77fa02e" containerID="cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f" exitCode=0 Jul 06 19:51:27.844121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.844099 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" Jul 06 19:51:27.844301 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.844119 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" event={"ID":"a247556b-2934-4525-8ec4-fa57d77fa02e","Type":"ContainerDied","Data":"cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f"} Jul 06 19:51:27.844301 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.844198 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn" event={"ID":"a247556b-2934-4525-8ec4-fa57d77fa02e","Type":"ContainerDied","Data":"7ab4920e5a831fd1290535dac6c4cd6d3eef428cc82cf9af9f554ec96ddca091"} Jul 06 19:51:27.844301 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.844221 2585 scope.go:117] "RemoveContainer" containerID="cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f" Jul 06 19:51:27.853142 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.853126 2585 scope.go:117] "RemoveContainer" containerID="cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f" Jul 06 19:51:27.853414 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:51:27.853393 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f\": container with ID starting with cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f not found: ID does not exist" containerID="cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f" Jul 06 19:51:27.853480 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.853421 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f"} err="failed to get container status \"cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f\": rpc error: code = NotFound desc = could not find container \"cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f\": container with ID starting with cdcbd14ed29c2f33bf58db613537fc53323b8602b5b5217ae304b1a82f20389f not found: ID does not exist" Jul 06 19:51:27.869209 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.869186 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn"] Jul 06 19:51:27.874269 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:27.874249 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-22b142fd-openshift-default-86bc96785f-8q5jn"] Jul 06 19:51:28.871866 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:51:28.871831 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a247556b-2934-4525-8ec4-fa57d77fa02e" path="/var/lib/kubelet/pods/a247556b-2934-4525-8ec4-fa57d77fa02e/volumes" Jul 06 19:53:08.000413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.000340 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8"] Jul 06 19:53:08.000845 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.000607 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a247556b-2934-4525-8ec4-fa57d77fa02e" containerName="istio-proxy" Jul 06 19:53:08.000845 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.000617 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a247556b-2934-4525-8ec4-fa57d77fa02e" containerName="istio-proxy" Jul 06 19:53:08.000845 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.000703 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a247556b-2934-4525-8ec4-fa57d77fa02e" containerName="istio-proxy" Jul 06 19:53:08.002522 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.002506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.007636 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.007610 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-925fb4a7-openshift-default-dockercfg-xw2b9\"" Jul 06 19:53:08.030733 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.030706 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8"] Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077638 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-podinfo\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-certs\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077777 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-data\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077814 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-token\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077853 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-envoy\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077907 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-credential-socket\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077943 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-socket\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.077973 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc060381-7b1b-4d42-85f5-7fd150d9a646-istiod-ca-cert\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.079466 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.078002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx5sh\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-kube-api-access-gx5sh\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178440 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178396 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-certs\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178457 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-data\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178483 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-token\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-envoy\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178571 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-credential-socket\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178623 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-socket\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178913 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc060381-7b1b-4d42-85f5-7fd150d9a646-istiod-ca-cert\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178913 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178650 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx5sh\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-kube-api-access-gx5sh\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178913 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-podinfo\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178913 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178862 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-certs\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.178913 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.178875 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-data\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.179163 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.179114 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-socket\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.179216 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.179181 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-credential-socket\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.179713 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.179688 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc060381-7b1b-4d42-85f5-7fd150d9a646-istiod-ca-cert\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.181031 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.181001 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-envoy\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.181309 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.181289 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-podinfo\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.187876 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.187846 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx5sh\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-kube-api-access-gx5sh\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.188122 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.188091 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-token\") pod \"e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.312034 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.311974 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:08.444307 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.444283 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8"] Jul 06 19:53:08.446011 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:53:08.445984 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc060381_7b1b_4d42_85f5_7fd150d9a646.slice/crio-120c548562e39bffdca23323182878eab95efd640646e499fb2dc45d99f576b6 WatchSource:0}: Error finding container 120c548562e39bffdca23323182878eab95efd640646e499fb2dc45d99f576b6: Status 404 returned error can't find the container with id 120c548562e39bffdca23323182878eab95efd640646e499fb2dc45d99f576b6 Jul 06 19:53:08.447965 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.447931 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:53:08.448050 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.447995 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:53:08.448050 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:08.448023 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:53:09.261980 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:09.261948 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" event={"ID":"dc060381-7b1b-4d42-85f5-7fd150d9a646","Type":"ContainerStarted","Data":"b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81"} Jul 06 19:53:09.262344 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:09.261996 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" event={"ID":"dc060381-7b1b-4d42-85f5-7fd150d9a646","Type":"ContainerStarted","Data":"120c548562e39bffdca23323182878eab95efd640646e499fb2dc45d99f576b6"} Jul 06 19:53:09.286285 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:09.286231 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" podStartSLOduration=2.2862151 podStartE2EDuration="2.2862151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:09.284297284 +0000 UTC m=+1798.926532113" watchObservedRunningTime="2026-07-06 19:53:09.2862151 +0000 UTC m=+1798.928449926" Jul 06 19:53:09.312468 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:09.312435 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:09.317294 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:09.317251 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:10.266753 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:10.266725 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:10.267719 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:10.267697 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:40.409239 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:40.409200 2585 patch_prober.go:28] interesting pod/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:57492->10.132.0.32:15021: read: connection reset by peer" start-of-body= Jul 06 19:53:40.409765 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:40.409281 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": read tcp 10.132.0.2:57492->10.132.0.32:15021: read: connection reset by peer" Jul 06 19:53:41.396455 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:41.396415 2585 generic.go:356] "Generic (PLEG): container finished" podID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerID="b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81" exitCode=0 Jul 06 19:53:41.396646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:41.396479 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" event={"ID":"dc060381-7b1b-4d42-85f5-7fd150d9a646","Type":"ContainerDied","Data":"b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81"} Jul 06 19:53:41.396911 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:41.396894 2585 scope.go:117] "RemoveContainer" containerID="b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81" Jul 06 19:53:41.397912 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:41.397878 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:53:41.398019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:41.397951 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:53:41.398019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:41.397989 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:53:42.402478 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:42.402443 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" event={"ID":"dc060381-7b1b-4d42-85f5-7fd150d9a646","Type":"ContainerStarted","Data":"f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be"} Jul 06 19:53:43.312860 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:43.312824 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:43.316371 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:43.316350 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:43.408972 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:43.408940 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:43.409943 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:43.409925 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:44.890508 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:44.890477 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8"] Jul 06 19:53:45.417536 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:45.417497 2585 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" secret="" err="secret \"e2e-mt-925fb4a7-openshift-default-dockercfg-xw2b9\" not found" Jul 06 19:53:46.421880 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:46.421841 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" containerID="cri-o://f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be" gracePeriod=30 Jul 06 19:53:51.452892 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:53:51.452863 2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc060381_7b1b_4d42_85f5_7fd150d9a646.slice/crio-conmon-f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be.scope\": RecentStats: unable to find data in memory cache]" Jul 06 19:53:51.453179 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:53:51.452969 2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc060381_7b1b_4d42_85f5_7fd150d9a646.slice/crio-conmon-f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be.scope\": RecentStats: unable to find data in memory cache]" Jul 06 19:53:51.661722 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.661697 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:51.845819 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845794 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-token\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.845990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845839 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx5sh\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-kube-api-access-gx5sh\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.845990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845863 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-data\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.845990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845877 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-socket\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.845990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845892 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc060381-7b1b-4d42-85f5-7fd150d9a646-istiod-ca-cert\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.845990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845914 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-certs\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.845990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845938 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-podinfo\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.845990 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.845979 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-envoy\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.846362 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.846042 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-credential-socket\") pod \"dc060381-7b1b-4d42-85f5-7fd150d9a646\" (UID: \"dc060381-7b1b-4d42-85f5-7fd150d9a646\") " Jul 06 19:53:51.846362 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.846154 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:51.846362 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.846285 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.846362 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.846347 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:51.846565 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.846369 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc060381-7b1b-4d42-85f5-7fd150d9a646-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:51.846565 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.846513 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:51.846894 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.846873 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:51.848140 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.848112 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:53:51.848237 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.848177 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:51.848393 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.848374 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-kube-api-access-gx5sh" (OuterVolumeSpecName: "kube-api-access-gx5sh") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "kube-api-access-gx5sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:51.848393 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.848389 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc060381-7b1b-4d42-85f5-7fd150d9a646" (UID: "dc060381-7b1b-4d42-85f5-7fd150d9a646"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:51.946822 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946788 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.946822 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946815 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx5sh\" (UniqueName: \"kubernetes.io/projected/dc060381-7b1b-4d42-85f5-7fd150d9a646-kube-api-access-gx5sh\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.946822 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946825 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.946822 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946833 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc060381-7b1b-4d42-85f5-7fd150d9a646-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.947071 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946841 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.947071 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946848 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.947071 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946856 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:51.947071 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:51.946863 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc060381-7b1b-4d42-85f5-7fd150d9a646-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:53:52.448646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.448612 2585 generic.go:356] "Generic (PLEG): container finished" podID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerID="f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be" exitCode=0 Jul 06 19:53:52.448829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.448679 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" Jul 06 19:53:52.448829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.448694 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" event={"ID":"dc060381-7b1b-4d42-85f5-7fd150d9a646","Type":"ContainerDied","Data":"f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be"} Jul 06 19:53:52.448829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.448774 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8" event={"ID":"dc060381-7b1b-4d42-85f5-7fd150d9a646","Type":"ContainerDied","Data":"120c548562e39bffdca23323182878eab95efd640646e499fb2dc45d99f576b6"} Jul 06 19:53:52.448829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.448804 2585 scope.go:117] "RemoveContainer" containerID="f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be" Jul 06 19:53:52.457841 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.457824 2585 scope.go:117] "RemoveContainer" containerID="b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81" Jul 06 19:53:52.465601 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.465586 2585 scope.go:117] "RemoveContainer" containerID="f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be" Jul 06 19:53:52.465901 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:53:52.465885 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be\": container with ID starting with f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be not found: ID does not exist" containerID="f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be" Jul 06 19:53:52.465950 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.465908 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be"} err="failed to get container status \"f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be\": rpc error: code = NotFound desc = could not find container \"f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be\": container with ID starting with f93777994339863c5a4b110a30b92c91dc45deaf25d9f7c05a35d9f40f01b6be not found: ID does not exist" Jul 06 19:53:52.465950 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.465925 2585 scope.go:117] "RemoveContainer" containerID="b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81" Jul 06 19:53:52.466130 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:53:52.466116 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81\": container with ID starting with b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81 not found: ID does not exist" containerID="b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81" Jul 06 19:53:52.466168 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.466133 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81"} err="failed to get container status \"b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81\": rpc error: code = NotFound desc = could not find container \"b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81\": container with ID starting with b7a9aee73ccd5393feb0ab95119aa89ccb6319d6fefddac5c85a9cf419dcbc81 not found: ID does not exist" Jul 06 19:53:52.470398 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.470378 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8"] Jul 06 19:53:52.476824 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.476803 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-925fb4a7-openshift-default-78c446bfc5-l4zs8"] Jul 06 19:53:52.871786 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:53:52.871760 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" path="/var/lib/kubelet/pods/dc060381-7b1b-4d42-85f5-7fd150d9a646/volumes" Jul 06 19:54:05.189933 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.189902 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7"] Jul 06 19:54:05.190370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.190193 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" Jul 06 19:54:05.190370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.190207 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" Jul 06 19:54:05.190370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.190219 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" Jul 06 19:54:05.190370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.190225 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" Jul 06 19:54:05.190370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.190287 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" Jul 06 19:54:05.193216 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.193198 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.195939 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.195922 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-08bc2a57-openshift-default-dockercfg-pz4zk\"" Jul 06 19:54:05.224953 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.224932 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7"] Jul 06 19:54:05.246693 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246645 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-credential-socket\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.246787 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246701 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-certs\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.246787 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246739 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-istio-token\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.246787 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246755 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5b8h\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-kube-api-access-b5b8h\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.246787 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246774 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07cc3c0-4946-4478-8250-942e8598b689-istiod-ca-cert\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.246994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246871 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-data\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.246994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246923 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-envoy\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.246994 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-socket\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.247116 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.246991 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07cc3c0-4946-4478-8250-942e8598b689-istio-podinfo\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348030 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.347992 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-istio-token\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348030 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348031 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5b8h\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-kube-api-access-b5b8h\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348278 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07cc3c0-4946-4478-8250-942e8598b689-istiod-ca-cert\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348278 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-data\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-envoy\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-socket\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07cc3c0-4946-4478-8250-942e8598b689-istio-podinfo\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348536 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348435 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-credential-socket\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348536 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348473 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-certs\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348700 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348681 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-data\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348778 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348734 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-socket\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348864 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348848 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-certs\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.348921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348847 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-credential-socket\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.349015 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.348994 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07cc3c0-4946-4478-8250-942e8598b689-istiod-ca-cert\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.350589 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.350572 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-envoy\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.350857 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.350842 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07cc3c0-4946-4478-8250-942e8598b689-istio-podinfo\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.357473 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.357450 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-istio-token\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.358512 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.358495 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5b8h\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-kube-api-access-b5b8h\") pod \"e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.503519 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.503495 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:05.640362 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.640326 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7"] Jul 06 19:54:05.641770 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:54:05.641746 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07cc3c0_4946_4478_8250_942e8598b689.slice/crio-a05a8746f907af6195a49fbfeb79c8aed5aedd2b3841c0a32b42919e9fe98287 WatchSource:0}: Error finding container a05a8746f907af6195a49fbfeb79c8aed5aedd2b3841c0a32b42919e9fe98287: Status 404 returned error can't find the container with id a05a8746f907af6195a49fbfeb79c8aed5aedd2b3841c0a32b42919e9fe98287 Jul 06 19:54:05.643816 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.643790 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:54:05.643892 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.643849 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:54:05.643892 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:05.643878 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:54:06.511145 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:06.511103 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" event={"ID":"d07cc3c0-4946-4478-8250-942e8598b689","Type":"ContainerStarted","Data":"1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239"} Jul 06 19:54:06.511524 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:06.511172 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" event={"ID":"d07cc3c0-4946-4478-8250-942e8598b689","Type":"ContainerStarted","Data":"a05a8746f907af6195a49fbfeb79c8aed5aedd2b3841c0a32b42919e9fe98287"} Jul 06 19:54:06.533506 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:06.533445 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" podStartSLOduration=1.533429531 podStartE2EDuration="1.533429531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:54:06.532979117 +0000 UTC m=+1856.175213959" watchObservedRunningTime="2026-07-06 19:54:06.533429531 +0000 UTC m=+1856.175664357" Jul 06 19:54:07.503644 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:07.503608 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:07.508613 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:07.508580 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:07.520001 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:07.518527 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:07.520802 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:07.520784 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:18.472318 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.472286 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g"] Jul 06 19:54:18.472818 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.472798 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc060381-7b1b-4d42-85f5-7fd150d9a646" containerName="istio-proxy" Jul 06 19:54:18.474752 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.474732 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.477900 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.477881 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-08bc2a57\"/\"e2e-lifecycle-model-08bc2a57-kserve-self-signed-certs\"" Jul 06 19:54:18.479181 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.479162 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-08bc2a57\"/\"kube-root-ca.crt\"" Jul 06 19:54:18.479291 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.479162 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-08bc2a57\"/\"openshift-service-ca.crt\"" Jul 06 19:54:18.480224 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.480205 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-08bc2a57\"/\"default-dockercfg-bz6t7\"" Jul 06 19:54:18.490195 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.490170 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g"] Jul 06 19:54:18.567555 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.567521 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711bb841-be48-4a9c-878d-b8a6f29758f3-tls-certs\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.567745 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.567627 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxvk\" (UniqueName: \"kubernetes.io/projected/711bb841-be48-4a9c-878d-b8a6f29758f3-kube-api-access-6fxvk\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.567745 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.567693 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-dshm\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.567849 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.567753 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-model-cache\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.567849 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.567805 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-home\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.567914 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.567849 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-tmp-dir\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.668813 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.668779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711bb841-be48-4a9c-878d-b8a6f29758f3-tls-certs\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.668851 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxvk\" (UniqueName: \"kubernetes.io/projected/711bb841-be48-4a9c-878d-b8a6f29758f3-kube-api-access-6fxvk\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.668890 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-dshm\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.668916 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-model-cache\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.668944 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-home\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669215 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.669036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-tmp-dir\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669432 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.669384 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-model-cache\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669613 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.669588 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-home\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.669736 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.669632 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-tmp-dir\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.671328 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.671304 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-dshm\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.671512 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.671496 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711bb841-be48-4a9c-878d-b8a6f29758f3-tls-certs\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.678545 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.678527 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxvk\" (UniqueName: \"kubernetes.io/projected/711bb841-be48-4a9c-878d-b8a6f29758f3-kube-api-access-6fxvk\") pod \"e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.784322 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.784286 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:18.955350 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:18.955321 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g"] Jul 06 19:54:18.958116 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:54:18.958086 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711bb841_be48_4a9c_878d_b8a6f29758f3.slice/crio-1a0c6cf37aa429fcaabec3b36325bc5847b7d518efb7410fa3df4a31c24836c1 WatchSource:0}: Error finding container 1a0c6cf37aa429fcaabec3b36325bc5847b7d518efb7410fa3df4a31c24836c1: Status 404 returned error can't find the container with id 1a0c6cf37aa429fcaabec3b36325bc5847b7d518efb7410fa3df4a31c24836c1 Jul 06 19:54:19.571885 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:19.571852 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" event={"ID":"711bb841-be48-4a9c-878d-b8a6f29758f3","Type":"ContainerStarted","Data":"d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db"} Jul 06 19:54:19.571885 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:19.571891 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" event={"ID":"711bb841-be48-4a9c-878d-b8a6f29758f3","Type":"ContainerStarted","Data":"1a0c6cf37aa429fcaabec3b36325bc5847b7d518efb7410fa3df4a31c24836c1"} Jul 06 19:54:19.597427 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:19.597378 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" podStartSLOduration=1.597364853 podStartE2EDuration="1.597364853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:54:19.597074956 +0000 UTC m=+1869.239309784" watchObservedRunningTime="2026-07-06 19:54:19.597364853 +0000 UTC m=+1869.239599678" Jul 06 19:54:23.218073 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.218026 2585 patch_prober.go:28] interesting pod/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:43114->10.132.0.33:15021: read: connection reset by peer" start-of-body= Jul 06 19:54:23.218531 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.218089 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:43114->10.132.0.33:15021: read: connection reset by peer" Jul 06 19:54:23.593199 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.593167 2585 generic.go:356] "Generic (PLEG): container finished" podID="d07cc3c0-4946-4478-8250-942e8598b689" containerID="1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239" exitCode=0 Jul 06 19:54:23.593390 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.593242 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" event={"ID":"d07cc3c0-4946-4478-8250-942e8598b689","Type":"ContainerDied","Data":"1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239"} Jul 06 19:54:23.593626 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.593613 2585 scope.go:117] "RemoveContainer" containerID="1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239" Jul 06 19:54:23.594552 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.594527 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:54:23.594607 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.594594 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:54:23.594646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:23.594630 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:54:24.599132 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:24.599099 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" event={"ID":"d07cc3c0-4946-4478-8250-942e8598b689","Type":"ContainerStarted","Data":"327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc"} Jul 06 19:54:25.504386 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:25.504352 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:25.504591 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:25.504570 2585 patch_prober.go:28] interesting pod/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 06 19:54:25.504698 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:25.504637 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 06 19:54:26.504847 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:26.504806 2585 patch_prober.go:28] interesting pod/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 06 19:54:26.505236 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:26.504880 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 06 19:54:27.503980 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:27.503946 2585 patch_prober.go:28] interesting pod/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 06 19:54:27.504169 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:27.504000 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 06 19:54:28.504028 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:28.503983 2585 patch_prober.go:28] interesting pod/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Jul 06 19:54:28.504425 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:28.504048 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Jul 06 19:54:28.784641 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:28.784557 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:28.784833 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:28.784789 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:28.797485 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:28.797461 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:54:29.507957 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:29.507920 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:29.508375 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:29.508183 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:29.508832 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:29.508815 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:54:29.632713 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:54:29.632684 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:55:01.704020 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:01.703936 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g"] Jul 06 19:55:16.723738 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:16.723698 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" podUID="711bb841-be48-4a9c-878d-b8a6f29758f3" containerName="main" containerID="cri-o://d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db" gracePeriod=45 Jul 06 19:55:16.970898 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:16.970875 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:55:17.051721 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.051695 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711bb841-be48-4a9c-878d-b8a6f29758f3-tls-certs\") pod \"711bb841-be48-4a9c-878d-b8a6f29758f3\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " Jul 06 19:55:17.051909 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.051777 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-model-cache\") pod \"711bb841-be48-4a9c-878d-b8a6f29758f3\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " Jul 06 19:55:17.051909 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.051804 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-home\") pod \"711bb841-be48-4a9c-878d-b8a6f29758f3\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " Jul 06 19:55:17.051909 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.051827 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fxvk\" (UniqueName: \"kubernetes.io/projected/711bb841-be48-4a9c-878d-b8a6f29758f3-kube-api-access-6fxvk\") pod \"711bb841-be48-4a9c-878d-b8a6f29758f3\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " Jul 06 19:55:17.051909 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.051857 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-dshm\") pod \"711bb841-be48-4a9c-878d-b8a6f29758f3\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " Jul 06 19:55:17.051909 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.051871 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-tmp-dir\") pod \"711bb841-be48-4a9c-878d-b8a6f29758f3\" (UID: \"711bb841-be48-4a9c-878d-b8a6f29758f3\") " Jul 06 19:55:17.052230 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.052051 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "711bb841-be48-4a9c-878d-b8a6f29758f3" (UID: "711bb841-be48-4a9c-878d-b8a6f29758f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:17.052230 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.052071 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-home" (OuterVolumeSpecName: "home") pod "711bb841-be48-4a9c-878d-b8a6f29758f3" (UID: "711bb841-be48-4a9c-878d-b8a6f29758f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:17.052230 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.052146 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "711bb841-be48-4a9c-878d-b8a6f29758f3" (UID: "711bb841-be48-4a9c-878d-b8a6f29758f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:17.053983 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.053960 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-dshm" (OuterVolumeSpecName: "dshm") pod "711bb841-be48-4a9c-878d-b8a6f29758f3" (UID: "711bb841-be48-4a9c-878d-b8a6f29758f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:17.053983 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.053968 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711bb841-be48-4a9c-878d-b8a6f29758f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "711bb841-be48-4a9c-878d-b8a6f29758f3" (UID: "711bb841-be48-4a9c-878d-b8a6f29758f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:55:17.054118 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.054052 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711bb841-be48-4a9c-878d-b8a6f29758f3-kube-api-access-6fxvk" (OuterVolumeSpecName: "kube-api-access-6fxvk") pod "711bb841-be48-4a9c-878d-b8a6f29758f3" (UID: "711bb841-be48-4a9c-878d-b8a6f29758f3"). InnerVolumeSpecName "kube-api-access-6fxvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:55:17.152641 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.152613 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:17.152641 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.152638 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:17.152829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.152647 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fxvk\" (UniqueName: \"kubernetes.io/projected/711bb841-be48-4a9c-878d-b8a6f29758f3-kube-api-access-6fxvk\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:17.152829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.152675 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:17.152829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.152684 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711bb841-be48-4a9c-878d-b8a6f29758f3-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:17.152829 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.152692 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711bb841-be48-4a9c-878d-b8a6f29758f3-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:17.832291 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.832248 2585 generic.go:356] "Generic (PLEG): container finished" podID="711bb841-be48-4a9c-878d-b8a6f29758f3" containerID="d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db" exitCode=0 Jul 06 19:55:17.832784 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.832297 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" Jul 06 19:55:17.832784 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.832296 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" event={"ID":"711bb841-be48-4a9c-878d-b8a6f29758f3","Type":"ContainerDied","Data":"d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db"} Jul 06 19:55:17.832784 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.832338 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g" event={"ID":"711bb841-be48-4a9c-878d-b8a6f29758f3","Type":"ContainerDied","Data":"1a0c6cf37aa429fcaabec3b36325bc5847b7d518efb7410fa3df4a31c24836c1"} Jul 06 19:55:17.832784 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.832357 2585 scope.go:117] "RemoveContainer" containerID="d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db" Jul 06 19:55:17.841022 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.841006 2585 scope.go:117] "RemoveContainer" containerID="d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db" Jul 06 19:55:17.841286 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:55:17.841251 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db\": container with ID starting with d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db not found: ID does not exist" containerID="d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db" Jul 06 19:55:17.841356 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.841293 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db"} err="failed to get container status \"d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db\": rpc error: code = NotFound desc = could not find container \"d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db\": container with ID starting with d27ab0d5b7b374ac0b485123953882b7fa38b0e9857447729a32ea5e39db45db not found: ID does not exist" Jul 06 19:55:17.854324 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.854302 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g"] Jul 06 19:55:17.858528 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:17.858508 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-08bc2a57/e2e-lifecycle-model-08bc2a57-kserve-9568b594c-d946g"] Jul 06 19:55:18.871335 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:18.871302 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711bb841-be48-4a9c-878d-b8a6f29758f3" path="/var/lib/kubelet/pods/711bb841-be48-4a9c-878d-b8a6f29758f3/volumes" Jul 06 19:55:46.531200 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:46.531156 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7"] Jul 06 19:55:46.531778 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:46.531431 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" containerID="cri-o://327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc" gracePeriod=30 Jul 06 19:55:51.787631 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.787603 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:55:51.862962 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.862865 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07cc3c0-4946-4478-8250-942e8598b689-istio-podinfo\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.862962 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.862923 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-certs\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.862962 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.862948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-credential-socket\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.863273 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863000 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5b8h\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-kube-api-access-b5b8h\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.863273 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863036 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-data\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.863273 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863069 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-envoy\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.863273 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863098 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-socket\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.863273 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863161 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07cc3c0-4946-4478-8250-942e8598b689-istiod-ca-cert\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.863273 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863188 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-istio-token\") pod \"d07cc3c0-4946-4478-8250-942e8598b689\" (UID: \"d07cc3c0-4946-4478-8250-942e8598b689\") " Jul 06 19:55:51.863555 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863273 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:51.863555 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863312 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:51.863555 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863466 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:51.863555 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863482 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.863555 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863501 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.863865 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.863610 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07cc3c0-4946-4478-8250-942e8598b689-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:55:51.864061 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.864032 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-data" (OuterVolumeSpecName: "istio-data") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:51.865397 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.865364 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-istio-token" (OuterVolumeSpecName: "istio-token") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:55:51.865397 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.865378 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d07cc3c0-4946-4478-8250-942e8598b689-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:55:51.865397 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.865362 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-kube-api-access-b5b8h" (OuterVolumeSpecName: "kube-api-access-b5b8h") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "kube-api-access-b5b8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:55:51.865566 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.865437 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d07cc3c0-4946-4478-8250-942e8598b689" (UID: "d07cc3c0-4946-4478-8250-942e8598b689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:55:51.964094 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.964056 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d07cc3c0-4946-4478-8250-942e8598b689-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.964094 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.964087 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.964094 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.964096 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d07cc3c0-4946-4478-8250-942e8598b689-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.964094 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.964105 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5b8h\" (UniqueName: \"kubernetes.io/projected/d07cc3c0-4946-4478-8250-942e8598b689-kube-api-access-b5b8h\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.964357 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.964115 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.964357 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.964124 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.964357 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.964132 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d07cc3c0-4946-4478-8250-942e8598b689-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:55:51.984908 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.984873 2585 generic.go:356] "Generic (PLEG): container finished" podID="d07cc3c0-4946-4478-8250-942e8598b689" containerID="327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc" exitCode=0 Jul 06 19:55:51.985070 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.984946 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" Jul 06 19:55:51.985070 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.984966 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" event={"ID":"d07cc3c0-4946-4478-8250-942e8598b689","Type":"ContainerDied","Data":"327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc"} Jul 06 19:55:51.985070 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.985036 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7" event={"ID":"d07cc3c0-4946-4478-8250-942e8598b689","Type":"ContainerDied","Data":"a05a8746f907af6195a49fbfeb79c8aed5aedd2b3841c0a32b42919e9fe98287"} Jul 06 19:55:51.985194 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.985069 2585 scope.go:117] "RemoveContainer" containerID="327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc" Jul 06 19:55:51.995002 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:51.994985 2585 scope.go:117] "RemoveContainer" containerID="1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239" Jul 06 19:55:52.004477 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:52.004451 2585 scope.go:117] "RemoveContainer" containerID="327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc" Jul 06 19:55:52.004849 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:55:52.004815 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc\": container with ID starting with 327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc not found: ID does not exist" containerID="327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc" Jul 06 19:55:52.004947 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:52.004860 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc"} err="failed to get container status \"327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc\": rpc error: code = NotFound desc = could not find container \"327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc\": container with ID starting with 327631827f50f0ed041cc5d144b634deef8bac5d0c3d4d26d82921299cfe81fc not found: ID does not exist" Jul 06 19:55:52.004947 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:52.004883 2585 scope.go:117] "RemoveContainer" containerID="1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239" Jul 06 19:55:52.005230 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:55:52.005208 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239\": container with ID starting with 1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239 not found: ID does not exist" containerID="1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239" Jul 06 19:55:52.005337 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:52.005236 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239"} err="failed to get container status \"1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239\": rpc error: code = NotFound desc = could not find container \"1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239\": container with ID starting with 1bcd6ee80dfe549ff076ecb9bb521c6f5a836e7760a637ebcf0bf08a76cf8239 not found: ID does not exist" Jul 06 19:55:52.014226 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:52.014193 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7"] Jul 06 19:55:52.016394 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:52.016370 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08bc2a57-openshift-default-d6d5b7f47-nrgl7"] Jul 06 19:55:52.872502 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:55:52.872464 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d07cc3c0-4946-4478-8250-942e8598b689" path="/var/lib/kubelet/pods/d07cc3c0-4946-4478-8250-942e8598b689/volumes" Jul 06 19:56:42.761390 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761356 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx"] Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761679 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761690 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761707 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761714 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761726 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711bb841-be48-4a9c-878d-b8a6f29758f3" containerName="main" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761732 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="711bb841-be48-4a9c-878d-b8a6f29758f3" containerName="main" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761778 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="711bb841-be48-4a9c-878d-b8a6f29758f3" containerName="main" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761787 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" Jul 06 19:56:42.761839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.761794 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d07cc3c0-4946-4478-8250-942e8598b689" containerName="istio-proxy" Jul 06 19:56:42.765403 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.765380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.768220 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.768198 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1e033f-openshift-default-dockercfg-wqm9p\"" Jul 06 19:56:42.777047 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.777024 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx"] Jul 06 19:56:42.790673 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790627 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-socket\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.790825 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790726 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istiod-ca-cert\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.790825 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-podinfo\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.790898 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790827 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-envoy\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.790932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgfq\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-kube-api-access-zfgfq\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.790932 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-certs\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.790996 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790971 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-data\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.790996 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.790989 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-credential-socket\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.791072 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.791016 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-token\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.891610 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.891549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-token\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.891870 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.891624 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-socket\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.891943 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.891900 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istiod-ca-cert\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.891943 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.891940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-podinfo\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892052 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892012 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-envoy\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892109 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgfq\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-kube-api-access-zfgfq\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892109 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892102 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-certs\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892118 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-socket\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892185 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-data\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892310 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-credential-socket\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892412 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892380 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-certs\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892571 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892546 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-credential-socket\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892727 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892588 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istiod-ca-cert\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.892727 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.892720 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-data\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.894346 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.894323 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-envoy\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.894467 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.894447 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-podinfo\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.901344 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.901319 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-token\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:42.901546 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:42.901529 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgfq\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-kube-api-access-zfgfq\") pod \"e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:43.076404 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:43.076303 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:43.215378 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:43.215353 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx"] Jul 06 19:56:43.218029 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:56:43.218003 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec417d9_32f7_45e0_820c_a3e81b9ac1e8.slice/crio-d2f91932b07cd714ce6354ae440e92c14e51fe561da3b4969d679ad7b6b717b4 WatchSource:0}: Error finding container d2f91932b07cd714ce6354ae440e92c14e51fe561da3b4969d679ad7b6b717b4: Status 404 returned error can't find the container with id d2f91932b07cd714ce6354ae440e92c14e51fe561da3b4969d679ad7b6b717b4 Jul 06 19:56:43.220088 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:43.220050 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:56:43.220212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:43.220138 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:56:43.220212 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:43.220181 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:56:44.218065 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:44.218025 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" event={"ID":"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8","Type":"ContainerStarted","Data":"1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d"} Jul 06 19:56:44.218437 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:44.218111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" event={"ID":"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8","Type":"ContainerStarted","Data":"d2f91932b07cd714ce6354ae440e92c14e51fe561da3b4969d679ad7b6b717b4"} Jul 06 19:56:44.244023 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:44.243972 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" podStartSLOduration=2.243956401 podStartE2EDuration="2.243956401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:44.242342788 +0000 UTC m=+2013.884577644" watchObservedRunningTime="2026-07-06 19:56:44.243956401 +0000 UTC m=+2013.886191226" Jul 06 19:56:45.076775 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:45.076733 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:45.081823 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:45.081797 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:45.222945 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:45.222915 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:45.223960 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:45.223940 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:56:51.340359 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.340328 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t"] Jul 06 19:56:51.343781 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.343757 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.346360 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.346334 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-64edc6-openshift-default-dockercfg-flr6k\"" Jul 06 19:56:51.357158 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.357131 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t"] Jul 06 19:56:51.463778 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.463698 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-podinfo\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.463778 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.463748 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-certs\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.464049 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.463784 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-token\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.464049 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.463899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istiod-ca-cert\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.464049 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.463969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpnpv\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-kube-api-access-mpnpv\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.464049 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.464030 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-credential-socket\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.464255 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.464057 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-data\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.464255 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.464114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-socket\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.464255 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.464146 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-envoy\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.564889 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.564845 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-token\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565100 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.564919 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istiod-ca-cert\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565100 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.564954 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnpv\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-kube-api-access-mpnpv\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565100 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565089 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-credential-socket\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565265 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-data\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565265 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-socket\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-envoy\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565433 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565405 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-podinfo\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565485 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565448 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-certs\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565539 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565494 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-data\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565596 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565551 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-credential-socket\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565694 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-socket\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565793 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565777 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-certs\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.565853 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.565832 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istiod-ca-cert\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.567475 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.567442 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-envoy\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.567704 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.567686 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-podinfo\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.580026 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.580003 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-token\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.581840 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.581821 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnpv\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-kube-api-access-mpnpv\") pod \"e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.655233 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.655129 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:51.800203 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.800179 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t"] Jul 06 19:56:51.802186 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:56:51.802157 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66ebf5b_c469_4f0f_bf76_aa7f2b77a31d.slice/crio-64a3e05d6fb2567db619709b655757de47b8f6ee8953580e8c02325445970c3d WatchSource:0}: Error finding container 64a3e05d6fb2567db619709b655757de47b8f6ee8953580e8c02325445970c3d: Status 404 returned error can't find the container with id 64a3e05d6fb2567db619709b655757de47b8f6ee8953580e8c02325445970c3d Jul 06 19:56:51.804321 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.804288 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:56:51.804411 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.804383 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:56:51.804454 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:51.804415 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:56:52.255083 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:52.255046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" event={"ID":"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d","Type":"ContainerStarted","Data":"a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99"} Jul 06 19:56:52.255269 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:52.255112 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" event={"ID":"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d","Type":"ContainerStarted","Data":"64a3e05d6fb2567db619709b655757de47b8f6ee8953580e8c02325445970c3d"} Jul 06 19:56:52.277866 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:52.277810 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" podStartSLOduration=1.277793642 podStartE2EDuration="1.277793642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:52.275863089 +0000 UTC m=+2021.918097917" watchObservedRunningTime="2026-07-06 19:56:52.277793642 +0000 UTC m=+2021.920028469" Jul 06 19:56:52.656258 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:52.656163 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:52.661186 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:52.661160 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:53.259877 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:53.259848 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:56:53.260904 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:56:53.260885 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:57:01.059007 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.057733 2585 patch_prober.go:28] interesting pod/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:33744->10.132.0.35:15021: read: connection reset by peer" start-of-body= Jul 06 19:57:01.059007 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.057818 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:33744->10.132.0.35:15021: read: connection reset by peer" Jul 06 19:57:01.295040 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.295003 2585 generic.go:356] "Generic (PLEG): container finished" podID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerID="1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d" exitCode=0 Jul 06 19:57:01.295217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.295076 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" event={"ID":"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8","Type":"ContainerDied","Data":"1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d"} Jul 06 19:57:01.295460 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.295447 2585 scope.go:117] "RemoveContainer" containerID="1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d" Jul 06 19:57:01.296440 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.296416 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:57:01.296493 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.296482 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:57:01.296530 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:01.296514 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:57:02.303410 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:02.303377 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" event={"ID":"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8","Type":"ContainerStarted","Data":"274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d"} Jul 06 19:57:03.077071 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:03.077032 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:57:03.077467 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:03.077439 2585 patch_prober.go:28] interesting pod/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 06 19:57:03.077690 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:03.077629 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 06 19:57:04.077567 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:04.077528 2585 patch_prober.go:28] interesting pod/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 06 19:57:04.078094 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:04.077597 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 06 19:57:05.077497 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:05.077463 2585 patch_prober.go:28] interesting pod/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 06 19:57:05.077640 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:05.077523 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 06 19:57:06.077331 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:06.077282 2585 patch_prober.go:28] interesting pod/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 06 19:57:06.077528 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:06.077352 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 06 19:57:07.080409 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:07.080379 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:57:07.080843 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:07.080758 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:57:07.081489 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:07.081471 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:57:08.261063 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.261024 2585 patch_prober.go:28] interesting pod/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:57:08.261489 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.261078 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:57:08.334802 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.334771 2585 generic.go:356] "Generic (PLEG): container finished" podID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerID="a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99" exitCode=0 Jul 06 19:57:08.334947 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.334843 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" event={"ID":"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d","Type":"ContainerDied","Data":"a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99"} Jul 06 19:57:08.335330 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.335314 2585 scope.go:117] "RemoveContainer" containerID="a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99" Jul 06 19:57:08.336297 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.336269 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:57:08.336369 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.336354 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:57:08.336416 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:08.336401 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:57:09.343506 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:09.343469 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" event={"ID":"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d","Type":"ContainerStarted","Data":"3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f"} Jul 06 19:57:09.655399 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:09.655298 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:57:09.655888 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:09.655846 2585 patch_prober.go:28] interesting pod/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:57:09.656021 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:09.655906 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:57:10.655628 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:10.655586 2585 patch_prober.go:28] interesting pod/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:57:10.656057 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:10.655685 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:57:11.655978 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:11.655937 2585 patch_prober.go:28] interesting pod/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:57:11.656387 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:11.656010 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:57:12.656239 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:12.656195 2585 patch_prober.go:28] interesting pod/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:57:12.656648 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:12.656268 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:57:13.659436 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:13.659402 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:57:13.659849 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:13.659796 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:57:13.660438 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:13.660417 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:57:24.633995 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.633963 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj"] Jul 06 19:57:24.636245 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.636229 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.640526 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.640506 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1e033f\"/\"test-model-1e033f-kserve-self-signed-certs\"" Jul 06 19:57:24.640526 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.640514 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1e033f\"/\"kube-root-ca.crt\"" Jul 06 19:57:24.640723 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.640530 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1e033f\"/\"openshift-service-ca.crt\"" Jul 06 19:57:24.640723 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.640506 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1e033f\"/\"default-dockercfg-q2fms\"" Jul 06 19:57:24.652905 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.652881 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj"] Jul 06 19:57:24.678571 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.678550 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-home\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.678708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.678584 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcq9\" (UniqueName: \"kubernetes.io/projected/3de3e9a8-2086-4a57-b58d-c0f78342099e-kube-api-access-ndcq9\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.678708 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.678615 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de3e9a8-2086-4a57-b58d-c0f78342099e-tls-certs\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.678840 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.678709 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-model-cache\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.678840 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.678745 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-dshm\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.678840 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.678779 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-tmp-dir\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.779761 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.779736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-dshm\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.779875 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.779773 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-tmp-dir\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.779875 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.779853 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-home\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.779972 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.779885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcq9\" (UniqueName: \"kubernetes.io/projected/3de3e9a8-2086-4a57-b58d-c0f78342099e-kube-api-access-ndcq9\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.779972 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.779934 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de3e9a8-2086-4a57-b58d-c0f78342099e-tls-certs\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.780050 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.779972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-model-cache\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.780489 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.780467 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-model-cache\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.781086 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.781066 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-tmp-dir\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.783019 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.782997 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-home\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.784280 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.784259 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-dshm\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.784644 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.784627 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de3e9a8-2086-4a57-b58d-c0f78342099e-tls-certs\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.792681 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.792634 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcq9\" (UniqueName: \"kubernetes.io/projected/3de3e9a8-2086-4a57-b58d-c0f78342099e-kube-api-access-ndcq9\") pod \"test-model-1e033f-kserve-6dcf76546f-n7msj\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:24.945493 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:24.945430 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:25.080562 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:25.080541 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj"] Jul 06 19:57:25.083240 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:57:25.083210 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de3e9a8_2086_4a57_b58d_c0f78342099e.slice/crio-3c21594d190edcc3e5206ec1a63cb2f853a79533747d0442f2f0c8d37bca8d8d WatchSource:0}: Error finding container 3c21594d190edcc3e5206ec1a63cb2f853a79533747d0442f2f0c8d37bca8d8d: Status 404 returned error can't find the container with id 3c21594d190edcc3e5206ec1a63cb2f853a79533747d0442f2f0c8d37bca8d8d Jul 06 19:57:25.417210 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:25.417177 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" event={"ID":"3de3e9a8-2086-4a57-b58d-c0f78342099e","Type":"ContainerStarted","Data":"ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c"} Jul 06 19:57:25.417210 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:25.417217 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" event={"ID":"3de3e9a8-2086-4a57-b58d-c0f78342099e","Type":"ContainerStarted","Data":"3c21594d190edcc3e5206ec1a63cb2f853a79533747d0442f2f0c8d37bca8d8d"} Jul 06 19:57:25.436670 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:25.436617 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" podStartSLOduration=1.436602475 podStartE2EDuration="1.436602475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:57:25.435801211 +0000 UTC m=+2055.078036037" watchObservedRunningTime="2026-07-06 19:57:25.436602475 +0000 UTC m=+2055.078837300" Jul 06 19:57:34.945895 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:34.945814 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:34.945895 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:34.945856 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:34.958222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:34.958195 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:57:35.474828 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:57:35.474795 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:58:28.954554 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:28.954517 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv"] Jul 06 19:58:28.958300 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:28.958278 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:28.962218 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:28.962196 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-64edc6\"/\"default-dockercfg-j5mtb\"" Jul 06 19:58:28.963319 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:28.963298 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-64edc6\"/\"kube-root-ca.crt\"" Jul 06 19:58:28.963319 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:28.963315 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-64edc6\"/\"test-model-64edc6-kserve-self-signed-certs\"" Jul 06 19:58:28.963484 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:28.963313 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-64edc6\"/\"openshift-service-ca.crt\"" Jul 06 19:58:29.047943 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.047908 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-home\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.048113 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.047965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-model-cache\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.048113 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.048038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-tmp-dir\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.048113 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.048078 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-dshm\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.048222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.048119 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkkw\" (UniqueName: \"kubernetes.io/projected/932a21ef-63c9-4a5c-b25d-481357d13bc0-kube-api-access-tjkkw\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.048222 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.048144 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932a21ef-63c9-4a5c-b25d-481357d13bc0-tls-certs\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149142 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149107 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-model-cache\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149157 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-tmp-dir\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149189 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-dshm\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkkw\" (UniqueName: \"kubernetes.io/projected/932a21ef-63c9-4a5c-b25d-481357d13bc0-kube-api-access-tjkkw\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149238 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932a21ef-63c9-4a5c-b25d-481357d13bc0-tls-certs\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149313 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-home\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149646 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149623 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-model-cache\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149743 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149687 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-tmp-dir\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.149743 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.149707 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-home\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.151488 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.151464 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-dshm\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.151635 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.151618 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932a21ef-63c9-4a5c-b25d-481357d13bc0-tls-certs\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:29.925605 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.925563 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv"] Jul 06 19:58:29.927942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:29.927908 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkkw\" (UniqueName: \"kubernetes.io/projected/932a21ef-63c9-4a5c-b25d-481357d13bc0-kube-api-access-tjkkw\") pod \"test-model-64edc6-kserve-6b4846d749-vc8gv\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:30.167502 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:30.167460 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:30.304185 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:30.304112 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv"] Jul 06 19:58:30.306378 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:58:30.306349 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod932a21ef_63c9_4a5c_b25d_481357d13bc0.slice/crio-4064ce42218d06b8035b2753162fd4cef7d916f3e4933dea4f82ed351e4b76a6 WatchSource:0}: Error finding container 4064ce42218d06b8035b2753162fd4cef7d916f3e4933dea4f82ed351e4b76a6: Status 404 returned error can't find the container with id 4064ce42218d06b8035b2753162fd4cef7d916f3e4933dea4f82ed351e4b76a6 Jul 06 19:58:30.705612 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:30.705576 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" event={"ID":"932a21ef-63c9-4a5c-b25d-481357d13bc0","Type":"ContainerStarted","Data":"278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616"} Jul 06 19:58:30.705612 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:30.705618 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" event={"ID":"932a21ef-63c9-4a5c-b25d-481357d13bc0","Type":"ContainerStarted","Data":"4064ce42218d06b8035b2753162fd4cef7d916f3e4933dea4f82ed351e4b76a6"} Jul 06 19:58:30.727726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:30.727633 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" podStartSLOduration=4.727619261 podStartE2EDuration="4.727619261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:58:30.725977794 +0000 UTC m=+2120.368212622" watchObservedRunningTime="2026-07-06 19:58:30.727619261 +0000 UTC m=+2120.369854121" Jul 06 19:58:40.167619 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:40.167566 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:40.167619 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:40.167633 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:40.180035 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:40.180013 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:40.764244 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:40.764210 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:58:55.882543 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:55.882510 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj"] Jul 06 19:58:57.110339 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:58:57.110303 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv"] Jul 06 19:59:10.902488 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:10.902381 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" podUID="3de3e9a8-2086-4a57-b58d-c0f78342099e" containerName="main" containerID="cri-o://ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c" gracePeriod=45 Jul 06 19:59:11.160626 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.160565 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:59:11.201029 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.200999 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-model-cache\") pod \"3de3e9a8-2086-4a57-b58d-c0f78342099e\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " Jul 06 19:59:11.201190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201086 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-home\") pod \"3de3e9a8-2086-4a57-b58d-c0f78342099e\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " Jul 06 19:59:11.201190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201105 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de3e9a8-2086-4a57-b58d-c0f78342099e-tls-certs\") pod \"3de3e9a8-2086-4a57-b58d-c0f78342099e\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " Jul 06 19:59:11.201190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201155 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-dshm\") pod \"3de3e9a8-2086-4a57-b58d-c0f78342099e\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " Jul 06 19:59:11.201190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201184 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-tmp-dir\") pod \"3de3e9a8-2086-4a57-b58d-c0f78342099e\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " Jul 06 19:59:11.201413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201209 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndcq9\" (UniqueName: \"kubernetes.io/projected/3de3e9a8-2086-4a57-b58d-c0f78342099e-kube-api-access-ndcq9\") pod \"3de3e9a8-2086-4a57-b58d-c0f78342099e\" (UID: \"3de3e9a8-2086-4a57-b58d-c0f78342099e\") " Jul 06 19:59:11.201413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201341 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3de3e9a8-2086-4a57-b58d-c0f78342099e" (UID: "3de3e9a8-2086-4a57-b58d-c0f78342099e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:11.201413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201359 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-home" (OuterVolumeSpecName: "home") pod "3de3e9a8-2086-4a57-b58d-c0f78342099e" (UID: "3de3e9a8-2086-4a57-b58d-c0f78342099e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:11.201563 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201485 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.201563 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201502 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.201563 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.201501 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3de3e9a8-2086-4a57-b58d-c0f78342099e" (UID: "3de3e9a8-2086-4a57-b58d-c0f78342099e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:11.203436 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.203402 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3e9a8-2086-4a57-b58d-c0f78342099e-kube-api-access-ndcq9" (OuterVolumeSpecName: "kube-api-access-ndcq9") pod "3de3e9a8-2086-4a57-b58d-c0f78342099e" (UID: "3de3e9a8-2086-4a57-b58d-c0f78342099e"). InnerVolumeSpecName "kube-api-access-ndcq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:11.203436 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.203419 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-dshm" (OuterVolumeSpecName: "dshm") pod "3de3e9a8-2086-4a57-b58d-c0f78342099e" (UID: "3de3e9a8-2086-4a57-b58d-c0f78342099e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:11.203573 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.203419 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de3e9a8-2086-4a57-b58d-c0f78342099e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3de3e9a8-2086-4a57-b58d-c0f78342099e" (UID: "3de3e9a8-2086-4a57-b58d-c0f78342099e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:59:11.302726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.302676 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de3e9a8-2086-4a57-b58d-c0f78342099e-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.302726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.302717 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.302726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.302727 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de3e9a8-2086-4a57-b58d-c0f78342099e-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.302726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.302738 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndcq9\" (UniqueName: \"kubernetes.io/projected/3de3e9a8-2086-4a57-b58d-c0f78342099e-kube-api-access-ndcq9\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.902039 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.901999 2585 generic.go:356] "Generic (PLEG): container finished" podID="3de3e9a8-2086-4a57-b58d-c0f78342099e" containerID="ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c" exitCode=0 Jul 06 19:59:11.902217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.902053 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" event={"ID":"3de3e9a8-2086-4a57-b58d-c0f78342099e","Type":"ContainerDied","Data":"ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c"} Jul 06 19:59:11.902217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.902063 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" Jul 06 19:59:11.902217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.902085 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj" event={"ID":"3de3e9a8-2086-4a57-b58d-c0f78342099e","Type":"ContainerDied","Data":"3c21594d190edcc3e5206ec1a63cb2f853a79533747d0442f2f0c8d37bca8d8d"} Jul 06 19:59:11.902217 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.902101 2585 scope.go:117] "RemoveContainer" containerID="ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c" Jul 06 19:59:11.910827 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.910809 2585 scope.go:117] "RemoveContainer" containerID="ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c" Jul 06 19:59:11.911120 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:59:11.911079 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c\": container with ID starting with ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c not found: ID does not exist" containerID="ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c" Jul 06 19:59:11.911120 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.911106 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c"} err="failed to get container status \"ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c\": rpc error: code = NotFound desc = could not find container \"ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c\": container with ID starting with ea3af93e9055635341303e49c8ed81e28cf20eb5f11e25237b34207f65d3225c not found: ID does not exist" Jul 06 19:59:11.928600 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.928577 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj"] Jul 06 19:59:11.932895 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:11.932871 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1e033f/test-model-1e033f-kserve-6dcf76546f-n7msj"] Jul 06 19:59:12.130732 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.130685 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" podUID="932a21ef-63c9-4a5c-b25d-481357d13bc0" containerName="main" containerID="cri-o://278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616" gracePeriod=45 Jul 06 19:59:12.395120 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.395099 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:59:12.515139 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515053 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932a21ef-63c9-4a5c-b25d-481357d13bc0-tls-certs\") pod \"932a21ef-63c9-4a5c-b25d-481357d13bc0\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " Jul 06 19:59:12.515139 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515095 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjkkw\" (UniqueName: \"kubernetes.io/projected/932a21ef-63c9-4a5c-b25d-481357d13bc0-kube-api-access-tjkkw\") pod \"932a21ef-63c9-4a5c-b25d-481357d13bc0\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " Jul 06 19:59:12.515139 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515138 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-dshm\") pod \"932a21ef-63c9-4a5c-b25d-481357d13bc0\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " Jul 06 19:59:12.515438 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515176 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-tmp-dir\") pod \"932a21ef-63c9-4a5c-b25d-481357d13bc0\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " Jul 06 19:59:12.515438 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515284 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-home\") pod \"932a21ef-63c9-4a5c-b25d-481357d13bc0\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " Jul 06 19:59:12.515438 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515310 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-model-cache\") pod \"932a21ef-63c9-4a5c-b25d-481357d13bc0\" (UID: \"932a21ef-63c9-4a5c-b25d-481357d13bc0\") " Jul 06 19:59:12.515607 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515525 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "932a21ef-63c9-4a5c-b25d-481357d13bc0" (UID: "932a21ef-63c9-4a5c-b25d-481357d13bc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.515607 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515532 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-home" (OuterVolumeSpecName: "home") pod "932a21ef-63c9-4a5c-b25d-481357d13bc0" (UID: "932a21ef-63c9-4a5c-b25d-481357d13bc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.515607 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515557 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "932a21ef-63c9-4a5c-b25d-481357d13bc0" (UID: "932a21ef-63c9-4a5c-b25d-481357d13bc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.515810 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515643 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.515810 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515687 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.515810 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.515698 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.517523 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.517496 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-dshm" (OuterVolumeSpecName: "dshm") pod "932a21ef-63c9-4a5c-b25d-481357d13bc0" (UID: "932a21ef-63c9-4a5c-b25d-481357d13bc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:12.517629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.517562 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932a21ef-63c9-4a5c-b25d-481357d13bc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "932a21ef-63c9-4a5c-b25d-481357d13bc0" (UID: "932a21ef-63c9-4a5c-b25d-481357d13bc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:59:12.517629 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.517577 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932a21ef-63c9-4a5c-b25d-481357d13bc0-kube-api-access-tjkkw" (OuterVolumeSpecName: "kube-api-access-tjkkw") pod "932a21ef-63c9-4a5c-b25d-481357d13bc0" (UID: "932a21ef-63c9-4a5c-b25d-481357d13bc0"). InnerVolumeSpecName "kube-api-access-tjkkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:12.617127 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.617089 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932a21ef-63c9-4a5c-b25d-481357d13bc0-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.617127 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.617119 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkkw\" (UniqueName: \"kubernetes.io/projected/932a21ef-63c9-4a5c-b25d-481357d13bc0-kube-api-access-tjkkw\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.617127 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.617130 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/932a21ef-63c9-4a5c-b25d-481357d13bc0-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:12.872929 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.872832 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de3e9a8-2086-4a57-b58d-c0f78342099e" path="/var/lib/kubelet/pods/3de3e9a8-2086-4a57-b58d-c0f78342099e/volumes" Jul 06 19:59:12.908661 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.908629 2585 generic.go:356] "Generic (PLEG): container finished" podID="932a21ef-63c9-4a5c-b25d-481357d13bc0" containerID="278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616" exitCode=0 Jul 06 19:59:12.908814 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.908696 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" Jul 06 19:59:12.908814 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.908726 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" event={"ID":"932a21ef-63c9-4a5c-b25d-481357d13bc0","Type":"ContainerDied","Data":"278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616"} Jul 06 19:59:12.908814 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.908771 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv" event={"ID":"932a21ef-63c9-4a5c-b25d-481357d13bc0","Type":"ContainerDied","Data":"4064ce42218d06b8035b2753162fd4cef7d916f3e4933dea4f82ed351e4b76a6"} Jul 06 19:59:12.908814 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.908795 2585 scope.go:117] "RemoveContainer" containerID="278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616" Jul 06 19:59:12.916749 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.916731 2585 scope.go:117] "RemoveContainer" containerID="278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616" Jul 06 19:59:12.917057 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:59:12.917026 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616\": container with ID starting with 278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616 not found: ID does not exist" containerID="278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616" Jul 06 19:59:12.917057 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.917048 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616"} err="failed to get container status \"278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616\": rpc error: code = NotFound desc = could not find container \"278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616\": container with ID starting with 278a8cd366ce26108166368f3211bbb3b11ec59e17a2bd2cdffa38b0cfb88616 not found: ID does not exist" Jul 06 19:59:12.926287 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.926258 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv"] Jul 06 19:59:12.931010 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:12.930991 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-64edc6/test-model-64edc6-kserve-6b4846d749-vc8gv"] Jul 06 19:59:14.872764 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:14.872720 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932a21ef-63c9-4a5c-b25d-481357d13bc0" path="/var/lib/kubelet/pods/932a21ef-63c9-4a5c-b25d-481357d13bc0/volumes" Jul 06 19:59:40.287358 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:40.287328 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx"] Jul 06 19:59:40.287790 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:40.287584 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" containerID="cri-o://274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d" gracePeriod=30 Jul 06 19:59:45.534858 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.534829 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:59:45.699308 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699222 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-certs\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699308 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699267 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istiod-ca-cert\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699308 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699296 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-podinfo\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699585 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699321 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-token\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699585 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699370 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-socket\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699585 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699394 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-data\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699585 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699427 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-credential-socket\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699585 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699477 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfgfq\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-kube-api-access-zfgfq\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699585 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699504 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-envoy\") pod \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\" (UID: \"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8\") " Jul 06 19:59:45.699585 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699532 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:45.699961 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699780 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.699961 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699772 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:45.699961 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699788 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:45.700132 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.699995 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:45.700379 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.700355 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:45.701518 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.701496 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:59:45.701667 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.701639 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-kube-api-access-zfgfq" (OuterVolumeSpecName: "kube-api-access-zfgfq") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "kube-api-access-zfgfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:45.701988 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.701966 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:45.702053 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.702012 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" (UID: "3ec417d9-32f7-45e0-820c-a3e81b9ac1e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:45.800453 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800421 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.800453 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800447 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.800453 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800458 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.800707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800467 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.800707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800475 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfgfq\" (UniqueName: \"kubernetes.io/projected/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-kube-api-access-zfgfq\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.800707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800483 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.800707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800491 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:45.800707 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:45.800498 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:46.057087 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.057050 2585 generic.go:356] "Generic (PLEG): container finished" podID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerID="274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d" exitCode=0 Jul 06 19:59:46.057241 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.057102 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" Jul 06 19:59:46.057241 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.057099 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" event={"ID":"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8","Type":"ContainerDied","Data":"274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d"} Jul 06 19:59:46.057241 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.057220 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx" event={"ID":"3ec417d9-32f7-45e0-820c-a3e81b9ac1e8","Type":"ContainerDied","Data":"d2f91932b07cd714ce6354ae440e92c14e51fe561da3b4969d679ad7b6b717b4"} Jul 06 19:59:46.057359 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.057251 2585 scope.go:117] "RemoveContainer" containerID="274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d" Jul 06 19:59:46.065797 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.065782 2585 scope.go:117] "RemoveContainer" containerID="1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d" Jul 06 19:59:46.073482 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.073462 2585 scope.go:117] "RemoveContainer" containerID="274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d" Jul 06 19:59:46.073734 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:59:46.073713 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d\": container with ID starting with 274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d not found: ID does not exist" containerID="274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d" Jul 06 19:59:46.073831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.073739 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d"} err="failed to get container status \"274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d\": rpc error: code = NotFound desc = could not find container \"274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d\": container with ID starting with 274b54b7fa5c3fe1c8dd021427bb10e6ed6998aaf4fe072febf84183d53d806d not found: ID does not exist" Jul 06 19:59:46.073831 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.073752 2585 scope.go:117] "RemoveContainer" containerID="1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d" Jul 06 19:59:46.073988 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:59:46.073973 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d\": container with ID starting with 1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d not found: ID does not exist" containerID="1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d" Jul 06 19:59:46.074027 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.073991 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d"} err="failed to get container status \"1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d\": rpc error: code = NotFound desc = could not find container \"1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d\": container with ID starting with 1468c7824327cd220b77975f934372131ed374358b4598c21108635a9093126d not found: ID does not exist" Jul 06 19:59:46.079889 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.079868 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx"] Jul 06 19:59:46.088245 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.088221 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1e033f-openshift-default-67f59f489c-g9mmx"] Jul 06 19:59:46.871718 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:46.871687 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" path="/var/lib/kubelet/pods/3ec417d9-32f7-45e0-820c-a3e81b9ac1e8/volumes" Jul 06 19:59:48.086726 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.086693 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t"] Jul 06 19:59:48.087097 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.086902 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" containerID="cri-o://3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f" gracePeriod=30 Jul 06 19:59:48.962702 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.962643 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n"] Jul 06 19:59:48.963017 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.962999 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" Jul 06 19:59:48.963105 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963019 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" Jul 06 19:59:48.963105 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963037 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932a21ef-63c9-4a5c-b25d-481357d13bc0" containerName="main" Jul 06 19:59:48.963105 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963046 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="932a21ef-63c9-4a5c-b25d-481357d13bc0" containerName="main" Jul 06 19:59:48.963105 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963078 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de3e9a8-2086-4a57-b58d-c0f78342099e" containerName="main" Jul 06 19:59:48.963105 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963086 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de3e9a8-2086-4a57-b58d-c0f78342099e" containerName="main" Jul 06 19:59:48.963356 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963179 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" Jul 06 19:59:48.963356 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963192 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" Jul 06 19:59:48.963356 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963203 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de3e9a8-2086-4a57-b58d-c0f78342099e" containerName="main" Jul 06 19:59:48.963356 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.963212 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="932a21ef-63c9-4a5c-b25d-481357d13bc0" containerName="main" Jul 06 19:59:48.966458 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.966437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:48.969500 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.969482 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c0c2c4-openshift-default-dockercfg-qq9kb\"" Jul 06 19:59:48.978382 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:48.978361 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n"] Jul 06 19:59:49.024591 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024566 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-podinfo\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024718 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-envoy\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024718 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024623 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-token\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024824 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024726 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-data\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024824 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024772 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vr2h\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-kube-api-access-8vr2h\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024824 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024817 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-credential-socket\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024984 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024839 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-socket\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024984 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024860 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-certs\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.024984 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.024879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d74118-6abe-452a-8466-cfa6f512bfe5-istiod-ca-cert\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125449 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125417 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-certs\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125449 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125449 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d74118-6abe-452a-8466-cfa6f512bfe5-istiod-ca-cert\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125480 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-podinfo\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-envoy\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-token\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-data\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125704 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vr2h\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-kube-api-access-8vr2h\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125785 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-credential-socket\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125810 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-socket\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.125967 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.125931 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-certs\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.126389 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.126039 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-data\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.126389 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.126106 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-socket\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.126389 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.126149 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-credential-socket\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.126389 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.126197 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d74118-6abe-452a-8466-cfa6f512bfe5-istiod-ca-cert\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.127979 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.127957 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-envoy\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.128289 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.128266 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-podinfo\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.142517 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.142480 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vr2h\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-kube-api-access-8vr2h\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.142650 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.142487 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-token\") pod \"e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.276546 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.276513 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:49.416686 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.416650 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n"] Jul 06 19:59:49.418461 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:59:49.418432 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d74118_6abe_452a_8466_cfa6f512bfe5.slice/crio-daa398258860a6e0536383c2e1f56d3f9c1f14d1ba57f92e1e639dc42f84542e WatchSource:0}: Error finding container daa398258860a6e0536383c2e1f56d3f9c1f14d1ba57f92e1e639dc42f84542e: Status 404 returned error can't find the container with id daa398258860a6e0536383c2e1f56d3f9c1f14d1ba57f92e1e639dc42f84542e Jul 06 19:59:49.420527 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.420498 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:59:49.420604 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.420560 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:59:49.420604 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:49.420588 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:59:50.078099 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:50.078060 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" event={"ID":"48d74118-6abe-452a-8466-cfa6f512bfe5","Type":"ContainerStarted","Data":"c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc"} Jul 06 19:59:50.078322 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:50.078124 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" event={"ID":"48d74118-6abe-452a-8466-cfa6f512bfe5","Type":"ContainerStarted","Data":"daa398258860a6e0536383c2e1f56d3f9c1f14d1ba57f92e1e639dc42f84542e"} Jul 06 19:59:50.102858 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:50.102804 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podStartSLOduration=2.102787406 podStartE2EDuration="2.102787406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:50.101587188 +0000 UTC m=+2199.743822006" watchObservedRunningTime="2026-07-06 19:59:50.102787406 +0000 UTC m=+2199.745022231" Jul 06 19:59:50.277739 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:50.277700 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:50.283178 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:50.283153 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:51.083165 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:51.083135 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:51.084121 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:51.084099 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 19:59:53.345474 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.345418 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:59:53.361370 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361345 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-certs\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361463 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361382 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpnpv\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-kube-api-access-mpnpv\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361463 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361411 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istiod-ca-cert\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361463 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361447 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-envoy\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361594 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361472 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-data\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361594 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361522 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-credential-socket\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361594 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361561 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-token\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361772 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361595 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-socket\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361772 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361628 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:53.361772 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361683 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-podinfo\") pod \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\" (UID: \"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d\") " Jul 06 19:59:53.361925 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361873 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:53.361988 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361914 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:53.361988 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361938 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:53.362096 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.361973 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.362096 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.362011 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.362373 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.362343 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:53.363751 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.363724 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:59:53.363751 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.363742 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-kube-api-access-mpnpv" (OuterVolumeSpecName: "kube-api-access-mpnpv") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "kube-api-access-mpnpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:53.363875 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.363771 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:53.363875 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.363791 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" (UID: "f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:59:53.463137 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.463099 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.463137 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.463128 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.463137 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.463137 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpnpv\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-kube-api-access-mpnpv\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.463137 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.463146 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.463413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.463155 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.463413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.463162 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:53.463413 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:53.463169 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 19:59:54.100486 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.100453 2585 generic.go:356] "Generic (PLEG): container finished" podID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerID="3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f" exitCode=0 Jul 06 19:59:54.100716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.100507 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" event={"ID":"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d","Type":"ContainerDied","Data":"3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f"} Jul 06 19:59:54.100716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.100552 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" event={"ID":"f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d","Type":"ContainerDied","Data":"64a3e05d6fb2567db619709b655757de47b8f6ee8953580e8c02325445970c3d"} Jul 06 19:59:54.100716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.100575 2585 scope.go:117] "RemoveContainer" containerID="3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f" Jul 06 19:59:54.100716 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.100508 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t" Jul 06 19:59:54.109600 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.109582 2585 scope.go:117] "RemoveContainer" containerID="a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99" Jul 06 19:59:54.117467 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.117449 2585 scope.go:117] "RemoveContainer" containerID="3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f" Jul 06 19:59:54.117730 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:59:54.117706 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f\": container with ID starting with 3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f not found: ID does not exist" containerID="3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f" Jul 06 19:59:54.117834 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.117737 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f"} err="failed to get container status \"3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f\": rpc error: code = NotFound desc = could not find container \"3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f\": container with ID starting with 3d5a0c684e5f762f36aee0099d5adde18519da4fad17e921a38882fe8fc7605f not found: ID does not exist" Jul 06 19:59:54.117834 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.117754 2585 scope.go:117] "RemoveContainer" containerID="a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99" Jul 06 19:59:54.117973 ip-10-0-131-106 kubenswrapper[2585]: E0706 19:59:54.117959 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99\": container with ID starting with a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99 not found: ID does not exist" containerID="a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99" Jul 06 19:59:54.118011 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.117977 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99"} err="failed to get container status \"a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99\": rpc error: code = NotFound desc = could not find container \"a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99\": container with ID starting with a5fa3efe389cabcc9c7b431917bbe4db8a5456b9f0af5a178f400a18836efb99 not found: ID does not exist" Jul 06 19:59:54.127127 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.127104 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t"] Jul 06 19:59:54.131633 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.131611 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-64edc6-openshift-default-6b788cbcf5-qmn4t"] Jul 06 19:59:54.872488 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:54.872445 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" path="/var/lib/kubelet/pods/f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d/volumes" Jul 06 19:59:57.244457 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.244427 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4"] Jul 06 19:59:57.244942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.244748 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" Jul 06 19:59:57.244942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.244765 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" Jul 06 19:59:57.244942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.244777 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" Jul 06 19:59:57.244942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.244783 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec417d9-32f7-45e0-820c-a3e81b9ac1e8" containerName="istio-proxy" Jul 06 19:59:57.244942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.244855 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" Jul 06 19:59:57.244942 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.244865 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" Jul 06 19:59:57.249280 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.249260 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.251811 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.251791 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bc0ca2-openshift-default-dockercfg-n9g6w\"" Jul 06 19:59:57.262041 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.262019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4"] Jul 06 19:59:57.300822 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.300788 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-socket\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.300996 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.300873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336fb1f9-72d5-42c2-ab7c-34e580474505-istiod-ca-cert\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.300996 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.300905 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-token\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.300996 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.300926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcfq\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-kube-api-access-nhcfq\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.300996 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.300955 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-credential-socket\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.301176 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.300999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-envoy\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.301176 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.301030 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-certs\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.301176 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.301107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-data\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.301176 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.301137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-podinfo\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.401968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.401933 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-certs\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.401968 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.401976 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-data\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.401993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-podinfo\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-socket\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336fb1f9-72d5-42c2-ab7c-34e580474505-istiod-ca-cert\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-token\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402099 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcfq\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-kube-api-access-nhcfq\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402150 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-credential-socket\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402190 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-envoy\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402609 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402377 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-certs\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402609 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402436 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-data\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402609 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402473 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-socket\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402609 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402562 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-credential-socket\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.402979 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.402963 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336fb1f9-72d5-42c2-ab7c-34e580474505-istiod-ca-cert\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.404302 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.404284 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-envoy\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.404595 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.404579 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-podinfo\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.410945 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.410922 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-token\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.411604 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.411581 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcfq\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-kube-api-access-nhcfq\") pod \"e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.559436 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.559352 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:57.714694 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.714667 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4"] Jul 06 19:59:57.717009 ip-10-0-131-106 kubenswrapper[2585]: W0706 19:59:57.716982 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336fb1f9_72d5_42c2_ab7c_34e580474505.slice/crio-7e6aa0d8b8c5460f5cadf8e0d34caa62ebb6b383979be34ceb51f1fb20b5eea1 WatchSource:0}: Error finding container 7e6aa0d8b8c5460f5cadf8e0d34caa62ebb6b383979be34ceb51f1fb20b5eea1: Status 404 returned error can't find the container with id 7e6aa0d8b8c5460f5cadf8e0d34caa62ebb6b383979be34ceb51f1fb20b5eea1 Jul 06 19:59:57.719305 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.719267 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:59:57.719385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.719338 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:59:57.719385 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:57.719364 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:59:58.121756 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:58.121718 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" event={"ID":"336fb1f9-72d5-42c2-ab7c-34e580474505","Type":"ContainerStarted","Data":"06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95"} Jul 06 19:59:58.121921 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:58.121778 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" event={"ID":"336fb1f9-72d5-42c2-ab7c-34e580474505","Type":"ContainerStarted","Data":"7e6aa0d8b8c5460f5cadf8e0d34caa62ebb6b383979be34ceb51f1fb20b5eea1"} Jul 06 19:59:58.145832 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:58.145782 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" podStartSLOduration=1.14576457 podStartE2EDuration="1.14576457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:58.143494244 +0000 UTC m=+2207.785729067" watchObservedRunningTime="2026-07-06 19:59:58.14576457 +0000 UTC m=+2207.787999395" Jul 06 19:59:58.559839 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:58.559795 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:58.564570 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:58.564545 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:59.126802 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:59.126768 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 19:59:59.127896 ip-10-0-131-106 kubenswrapper[2585]: I0706 19:59:59.127871 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 20:00:06.084496 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.084458 2585 patch_prober.go:28] interesting pod/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 06 20:00:06.084931 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.084517 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 06 20:00:06.164456 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.164424 2585 generic.go:356] "Generic (PLEG): container finished" podID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerID="c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc" exitCode=0 Jul 06 20:00:06.164625 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.164471 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" event={"ID":"48d74118-6abe-452a-8466-cfa6f512bfe5","Type":"ContainerDied","Data":"c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc"} Jul 06 20:00:06.164872 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.164857 2585 scope.go:117] "RemoveContainer" containerID="c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc" Jul 06 20:00:06.165848 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.165819 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:06.165937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.165888 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:06.165937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:06.165927 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:07.173827 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:07.173787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" event={"ID":"48d74118-6abe-452a-8466-cfa6f512bfe5","Type":"ContainerStarted","Data":"aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a"} Jul 06 20:00:07.276899 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:07.276859 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 20:00:07.277206 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:07.277182 2585 patch_prober.go:28] interesting pod/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 06 20:00:07.277292 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:07.277235 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 06 20:00:08.277783 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:08.277743 2585 patch_prober.go:28] interesting pod/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 06 20:00:08.278197 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:08.277802 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 06 20:00:09.277030 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:09.276989 2585 patch_prober.go:28] interesting pod/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 06 20:00:09.277237 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:09.277060 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 06 20:00:10.277491 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:10.277441 2585 patch_prober.go:28] interesting pod/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 06 20:00:10.278038 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:10.277513 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 06 20:00:11.277745 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:11.277623 2585 patch_prober.go:28] interesting pod/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 06 20:00:11.277745 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:11.277724 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 06 20:00:12.281412 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:12.281378 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 20:00:12.281858 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:12.281628 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 20:00:12.282322 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:12.282304 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 20:00:15.167831 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:15.167789 2585 patch_prober.go:28] interesting pod/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 20:00:15.168219 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:15.167845 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:00:16.216784 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:16.216649 2585 generic.go:356] "Generic (PLEG): container finished" podID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerID="06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95" exitCode=0 Jul 06 20:00:16.216784 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:16.216728 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" event={"ID":"336fb1f9-72d5-42c2-ab7c-34e580474505","Type":"ContainerDied","Data":"06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95"} Jul 06 20:00:16.217362 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:16.217195 2585 scope.go:117] "RemoveContainer" containerID="06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95" Jul 06 20:00:16.218381 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:16.218345 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:16.218518 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:16.218429 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:16.218518 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:16.218474 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:17.223396 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:17.223362 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" event={"ID":"336fb1f9-72d5-42c2-ab7c-34e580474505","Type":"ContainerStarted","Data":"c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af"} Jul 06 20:00:17.559841 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:17.559806 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 20:00:17.560129 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:17.560105 2585 patch_prober.go:28] interesting pod/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 20:00:17.560227 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:17.560179 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 20:00:18.559782 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:18.559747 2585 patch_prober.go:28] interesting pod/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 20:00:18.560213 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:18.559802 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 20:00:19.443689 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:19.443522 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n"] Jul 06 20:00:19.444682 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:19.444047 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" containerID="cri-o://aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a" gracePeriod=30 Jul 06 20:00:19.560009 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:19.559967 2585 patch_prober.go:28] interesting pod/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 20:00:19.560494 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:19.560040 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 20:00:20.559848 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:20.559791 2585 patch_prober.go:28] interesting pod/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 06 20:00:20.560458 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:20.559873 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 06 20:00:21.563258 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:21.563225 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 20:00:21.563801 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:21.563610 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 20:00:21.564431 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:21.564409 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 20:00:24.707738 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.707714 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 20:00:24.860479 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860448 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-credential-socket\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.860671 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860495 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vr2h\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-kube-api-access-8vr2h\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.860671 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860525 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-token\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.860671 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860539 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-envoy\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.860671 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860576 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-podinfo\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.860671 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860603 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-socket\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.860671 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860639 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-data\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.861000 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860686 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-certs\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.861000 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860717 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d74118-6abe-452a-8466-cfa6f512bfe5-istiod-ca-cert\") pod \"48d74118-6abe-452a-8466-cfa6f512bfe5\" (UID: \"48d74118-6abe-452a-8466-cfa6f512bfe5\") " Jul 06 20:00:24.861000 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860789 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:24.861000 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.860992 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:24.861205 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.861045 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.861205 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.861083 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d74118-6abe-452a-8466-cfa6f512bfe5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:00:24.861299 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.861234 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:24.861970 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.861946 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-data" (OuterVolumeSpecName: "istio-data") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:24.862962 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.862926 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-kube-api-access-8vr2h" (OuterVolumeSpecName: "kube-api-access-8vr2h") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "kube-api-access-8vr2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:24.862962 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.862952 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:00:24.863194 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.863178 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:24.863252 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.863204 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-token" (OuterVolumeSpecName: "istio-token") pod "48d74118-6abe-452a-8466-cfa6f512bfe5" (UID: "48d74118-6abe-452a-8466-cfa6f512bfe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:24.909380 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:00:24.909338 2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d74118_6abe_452a_8466_cfa6f512bfe5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d74118_6abe_452a_8466_cfa6f512bfe5.slice/crio-daa398258860a6e0536383c2e1f56d3f9c1f14d1ba57f92e1e639dc42f84542e\": RecentStats: unable to find data in memory cache]" Jul 06 20:00:24.962264 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962233 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vr2h\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-kube-api-access-8vr2h\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.962264 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962260 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.962264 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962269 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.962264 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962276 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.962544 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962284 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.962544 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962292 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.962544 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962301 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48d74118-6abe-452a-8466-cfa6f512bfe5-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:24.962544 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:24.962308 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48d74118-6abe-452a-8466-cfa6f512bfe5-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:25.263272 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.263236 2585 generic.go:356] "Generic (PLEG): container finished" podID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerID="aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a" exitCode=0 Jul 06 20:00:25.263445 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.263285 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" Jul 06 20:00:25.263445 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.263323 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" event={"ID":"48d74118-6abe-452a-8466-cfa6f512bfe5","Type":"ContainerDied","Data":"aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a"} Jul 06 20:00:25.263445 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.263381 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n" event={"ID":"48d74118-6abe-452a-8466-cfa6f512bfe5","Type":"ContainerDied","Data":"daa398258860a6e0536383c2e1f56d3f9c1f14d1ba57f92e1e639dc42f84542e"} Jul 06 20:00:25.263445 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.263404 2585 scope.go:117] "RemoveContainer" containerID="aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a" Jul 06 20:00:25.271842 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.271824 2585 scope.go:117] "RemoveContainer" containerID="c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc" Jul 06 20:00:25.279193 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.279173 2585 scope.go:117] "RemoveContainer" containerID="aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a" Jul 06 20:00:25.279430 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:00:25.279409 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a\": container with ID starting with aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a not found: ID does not exist" containerID="aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a" Jul 06 20:00:25.279493 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.279442 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a"} err="failed to get container status \"aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a\": rpc error: code = NotFound desc = could not find container \"aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a\": container with ID starting with aa89d35512af77d430f4db144f42880dd8cd3fd9d6423b2977189aad3ea44a4a not found: ID does not exist" Jul 06 20:00:25.279493 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.279460 2585 scope.go:117] "RemoveContainer" containerID="c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc" Jul 06 20:00:25.279704 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:00:25.279687 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc\": container with ID starting with c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc not found: ID does not exist" containerID="c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc" Jul 06 20:00:25.279763 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.279709 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc"} err="failed to get container status \"c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc\": rpc error: code = NotFound desc = could not find container \"c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc\": container with ID starting with c0c1e842d8813da4cfea338763ccd9951f5fbb2d626164864c9f0da97bd397fc not found: ID does not exist" Jul 06 20:00:25.284595 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.284572 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n"] Jul 06 20:00:25.292209 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:25.292189 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c0c2c4-openshift-default-679b6b7cbf-xwh9n"] Jul 06 20:00:26.822925 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:26.822888 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4"] Jul 06 20:00:26.823309 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:26.823182 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" containerID="cri-o://c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af" gracePeriod=30 Jul 06 20:00:26.872225 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:26.872193 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" path="/var/lib/kubelet/pods/48d74118-6abe-452a-8466-cfa6f512bfe5/volumes" Jul 06 20:00:27.684957 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.684922 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp"] Jul 06 20:00:27.685241 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685228 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" Jul 06 20:00:27.685286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685243 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" Jul 06 20:00:27.685286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685259 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" Jul 06 20:00:27.685286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685265 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" Jul 06 20:00:27.685286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685277 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" Jul 06 20:00:27.685286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685286 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66ebf5b-c469-4f0f-bf76-aa7f2b77a31d" containerName="istio-proxy" Jul 06 20:00:27.685452 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685342 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" Jul 06 20:00:27.685452 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.685354 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d74118-6abe-452a-8466-cfa6f512bfe5" containerName="istio-proxy" Jul 06 20:00:27.688490 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.688470 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.691341 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.691316 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e1f716-openshift-default-dockercfg-w7jdm\"" Jul 06 20:00:27.700705 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.700678 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp"] Jul 06 20:00:27.790034 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.789990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-data\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790240 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790072 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-envoy\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790240 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-credential-socket\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790240 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790154 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-istio-token\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790240 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-socket\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790240 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790216 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b18423-503b-4879-9763-329386c91c3a-istio-podinfo\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790512 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbfv\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-kube-api-access-2tbfv\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790512 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790292 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b18423-503b-4879-9763-329386c91c3a-istiod-ca-cert\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.790512 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.790316 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-certs\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891030 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.890987 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b18423-503b-4879-9763-329386c91c3a-istio-podinfo\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891042 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbfv\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-kube-api-access-2tbfv\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891077 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b18423-503b-4879-9763-329386c91c3a-istiod-ca-cert\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-certs\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891135 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-data\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891192 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-envoy\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891253 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-credential-socket\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891283 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-istio-token\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891545 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891313 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-socket\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891975 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891587 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-data\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891975 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891623 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-certs\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.891975 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891772 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-socket\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.892095 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.891985 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-credential-socket\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.892095 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.892036 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b18423-503b-4879-9763-329386c91c3a-istiod-ca-cert\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.893563 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.893545 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b18423-503b-4879-9763-329386c91c3a-istio-podinfo\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.893628 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.893551 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-envoy\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.899280 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.899254 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-istio-token\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.899492 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.899472 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbfv\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-kube-api-access-2tbfv\") pod \"e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:27.998332 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:27.998301 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:28.134238 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.134073 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp"] Jul 06 20:00:28.136240 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:00:28.136209 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b18423_503b_4879_9763_329386c91c3a.slice/crio-6baee27271365696790ea7ef49fabb479299131d738b62ce0abc2538554530fd WatchSource:0}: Error finding container 6baee27271365696790ea7ef49fabb479299131d738b62ce0abc2538554530fd: Status 404 returned error can't find the container with id 6baee27271365696790ea7ef49fabb479299131d738b62ce0abc2538554530fd Jul 06 20:00:28.138492 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.138459 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:28.138564 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.138532 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:28.138564 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.138560 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:28.280147 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.280019 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" event={"ID":"86b18423-503b-4879-9763-329386c91c3a","Type":"ContainerStarted","Data":"d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af"} Jul 06 20:00:28.280147 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.280078 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" event={"ID":"86b18423-503b-4879-9763-329386c91c3a","Type":"ContainerStarted","Data":"6baee27271365696790ea7ef49fabb479299131d738b62ce0abc2538554530fd"} Jul 06 20:00:28.302760 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.302697 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podStartSLOduration=1.302677378 podStartE2EDuration="1.302677378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:28.300754601 +0000 UTC m=+2237.942989428" watchObservedRunningTime="2026-07-06 20:00:28.302677378 +0000 UTC m=+2237.944912204" Jul 06 20:00:28.998904 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:28.998871 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:30.004035 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:30.004002 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:30.289729 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:30.289629 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:30.290670 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:30.290635 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:32.079362 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.079334 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 20:00:32.227914 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.227821 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-socket\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.227914 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.227884 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhcfq\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-kube-api-access-nhcfq\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.227914 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.227911 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-envoy\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.228206 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228089 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:32.228206 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228126 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-token\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.228206 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228166 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-credential-socket\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.228206 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228192 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-certs\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.228415 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228218 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-data\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.228415 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228240 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-podinfo\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.228415 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228277 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336fb1f9-72d5-42c2-ab7c-34e580474505-istiod-ca-cert\") pod \"336fb1f9-72d5-42c2-ab7c-34e580474505\" (UID: \"336fb1f9-72d5-42c2-ab7c-34e580474505\") " Jul 06 20:00:32.228565 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228483 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:32.228565 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228495 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:32.228699 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228594 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.228699 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228692 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.228802 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228711 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.228802 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.228789 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336fb1f9-72d5-42c2-ab7c-34e580474505-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:00:32.229524 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.229492 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-data" (OuterVolumeSpecName: "istio-data") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:32.230277 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.230249 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:32.230377 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.230316 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-kube-api-access-nhcfq" (OuterVolumeSpecName: "kube-api-access-nhcfq") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "kube-api-access-nhcfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:32.230421 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.230375 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-token" (OuterVolumeSpecName: "istio-token") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:32.230751 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.230737 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "336fb1f9-72d5-42c2-ab7c-34e580474505" (UID: "336fb1f9-72d5-42c2-ab7c-34e580474505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:00:32.299589 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.299560 2585 generic.go:356] "Generic (PLEG): container finished" podID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerID="c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af" exitCode=0 Jul 06 20:00:32.299701 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.299614 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" Jul 06 20:00:32.299701 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.299645 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" event={"ID":"336fb1f9-72d5-42c2-ab7c-34e580474505","Type":"ContainerDied","Data":"c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af"} Jul 06 20:00:32.299781 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.299717 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4" event={"ID":"336fb1f9-72d5-42c2-ab7c-34e580474505","Type":"ContainerDied","Data":"7e6aa0d8b8c5460f5cadf8e0d34caa62ebb6b383979be34ceb51f1fb20b5eea1"} Jul 06 20:00:32.299781 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.299739 2585 scope.go:117] "RemoveContainer" containerID="c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af" Jul 06 20:00:32.307778 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.307764 2585 scope.go:117] "RemoveContainer" containerID="06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95" Jul 06 20:00:32.314724 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.314703 2585 scope.go:117] "RemoveContainer" containerID="c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af" Jul 06 20:00:32.314996 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:00:32.314974 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af\": container with ID starting with c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af not found: ID does not exist" containerID="c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af" Jul 06 20:00:32.315059 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.315005 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af"} err="failed to get container status \"c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af\": rpc error: code = NotFound desc = could not find container \"c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af\": container with ID starting with c8d2f1f562dc5af0652637b0497ab017efe9f2ff439d3a5269f2e3d919a4a0af not found: ID does not exist" Jul 06 20:00:32.315059 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.315024 2585 scope.go:117] "RemoveContainer" containerID="06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95" Jul 06 20:00:32.315300 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:00:32.315277 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95\": container with ID starting with 06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95 not found: ID does not exist" containerID="06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95" Jul 06 20:00:32.315346 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.315305 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95"} err="failed to get container status \"06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95\": rpc error: code = NotFound desc = could not find container \"06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95\": container with ID starting with 06b2629c4ffffe0706353a51292512f97fa7f91dbed7d355422b3bef7035cd95 not found: ID does not exist" Jul 06 20:00:32.329284 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.329263 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4"] Jul 06 20:00:32.329577 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.329559 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhcfq\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-kube-api-access-nhcfq\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.329624 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.329581 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.329624 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.329590 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.329624 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.329601 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.329624 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.329613 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336fb1f9-72d5-42c2-ab7c-34e580474505-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.329624 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.329621 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336fb1f9-72d5-42c2-ab7c-34e580474505-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:00:32.334148 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.334128 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bc0ca2-openshift-default-67d6ff9455-x5cq4"] Jul 06 20:00:32.871829 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:32.871799 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" path="/var/lib/kubelet/pods/336fb1f9-72d5-42c2-ab7c-34e580474505/volumes" Jul 06 20:00:35.289087 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.289049 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m"] Jul 06 20:00:35.289508 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.289343 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" Jul 06 20:00:35.289508 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.289354 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" Jul 06 20:00:35.289508 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.289392 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" Jul 06 20:00:35.289508 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.289398 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" Jul 06 20:00:35.289508 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.289445 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" Jul 06 20:00:35.289508 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.289452 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="336fb1f9-72d5-42c2-ab7c-34e580474505" containerName="istio-proxy" Jul 06 20:00:35.292988 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.292969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.296196 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.296174 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-035267-openshift-default-dockercfg-n57m2\"" Jul 06 20:00:35.307131 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.307109 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m"] Jul 06 20:00:35.349632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-credential-socket\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349756 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349643 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-data\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349756 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349680 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-socket\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349756 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349742 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istiod-ca-cert\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349864 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-token\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349864 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349821 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-envoy\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349864 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349843 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-podinfo\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349957 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349865 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpd4v\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-kube-api-access-wpd4v\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.349957 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.349883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-certs\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450356 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450314 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-token\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450546 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-envoy\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450546 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450401 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-podinfo\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450546 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450426 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpd4v\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-kube-api-access-wpd4v\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450546 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-certs\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450546 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450486 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-credential-socket\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450546 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450523 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-data\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450546 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-socket\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.450949 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.450578 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istiod-ca-cert\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.451063 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.451036 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-credential-socket\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.451192 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.451112 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-certs\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.451192 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.451154 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-data\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.451192 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.451184 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-socket\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.451310 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.451296 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istiod-ca-cert\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.453182 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.453164 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-envoy\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.453415 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.453397 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-podinfo\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.473304 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.473272 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-token\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.473441 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.473412 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpd4v\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-kube-api-access-wpd4v\") pod \"e2e-shared-b-035267-openshift-default-999cd8f55-5px2m\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.602677 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.602572 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:35.737567 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.737543 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m"] Jul 06 20:00:35.739408 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:00:35.739381 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97b7aa8_0cc5_4c5c_983a_74607a28013e.slice/crio-dceebc2e3ac7fa78e9eea022beb918c0f08f095f127b8b8e97717a1682d1f49f WatchSource:0}: Error finding container dceebc2e3ac7fa78e9eea022beb918c0f08f095f127b8b8e97717a1682d1f49f: Status 404 returned error can't find the container with id dceebc2e3ac7fa78e9eea022beb918c0f08f095f127b8b8e97717a1682d1f49f Jul 06 20:00:35.741385 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.741353 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:35.741478 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.741416 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:35.741478 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:35.741445 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:36.320358 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:36.320320 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" event={"ID":"c97b7aa8-0cc5-4c5c-983a-74607a28013e","Type":"ContainerStarted","Data":"2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836"} Jul 06 20:00:36.320860 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:36.320374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" event={"ID":"c97b7aa8-0cc5-4c5c-983a-74607a28013e","Type":"ContainerStarted","Data":"dceebc2e3ac7fa78e9eea022beb918c0f08f095f127b8b8e97717a1682d1f49f"} Jul 06 20:00:36.344528 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:36.344480 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" podStartSLOduration=1.344467574 podStartE2EDuration="1.344467574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:36.342192592 +0000 UTC m=+2245.984427444" watchObservedRunningTime="2026-07-06 20:00:36.344467574 +0000 UTC m=+2245.986702400" Jul 06 20:00:36.603810 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:36.603727 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:36.608608 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:36.608584 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:37.324373 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:37.324347 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:37.325210 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:37.325190 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:45.291605 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.291563 2585 patch_prober.go:28] interesting pod/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 06 20:00:45.292075 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.291675 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 06 20:00:45.371986 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.371954 2585 generic.go:356] "Generic (PLEG): container finished" podID="86b18423-503b-4879-9763-329386c91c3a" containerID="d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af" exitCode=0 Jul 06 20:00:45.372156 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.372031 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" event={"ID":"86b18423-503b-4879-9763-329386c91c3a","Type":"ContainerDied","Data":"d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af"} Jul 06 20:00:45.372457 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.372444 2585 scope.go:117] "RemoveContainer" containerID="d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af" Jul 06 20:00:45.373519 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.373488 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:45.373627 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.373577 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:45.373627 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:45.373619 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:46.381779 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:46.381744 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" event={"ID":"86b18423-503b-4879-9763-329386c91c3a","Type":"ContainerStarted","Data":"cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2"} Jul 06 20:00:46.998593 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:46.998551 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:46.998915 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:46.998885 2585 patch_prober.go:28] interesting pod/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 06 20:00:46.998980 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:46.998957 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 06 20:00:47.999666 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:47.999621 2585 patch_prober.go:28] interesting pod/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 06 20:00:48.000053 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:47.999740 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 06 20:00:48.998939 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:48.998903 2585 patch_prober.go:28] interesting pod/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 06 20:00:48.999136 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:48.998957 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 06 20:00:49.999403 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:49.999368 2585 patch_prober.go:28] interesting pod/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 06 20:00:49.999844 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:49.999427 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 06 20:00:50.401998 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:50.401911 2585 generic.go:356] "Generic (PLEG): container finished" podID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerID="2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836" exitCode=0 Jul 06 20:00:50.401998 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:50.401963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" event={"ID":"c97b7aa8-0cc5-4c5c-983a-74607a28013e","Type":"ContainerDied","Data":"2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836"} Jul 06 20:00:50.402387 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:50.402372 2585 scope.go:117] "RemoveContainer" containerID="2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836" Jul 06 20:00:50.403484 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:50.403454 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:50.403570 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:50.403541 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:50.403611 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:50.403598 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 20:00:51.003005 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:51.002974 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:51.003384 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:51.003361 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:51.004044 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:51.004025 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:00:51.411013 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:51.410878 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" event={"ID":"c97b7aa8-0cc5-4c5c-983a-74607a28013e","Type":"ContainerStarted","Data":"04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047"} Jul 06 20:00:51.603191 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:51.603153 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:51.604159 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:51.604131 2585 patch_prober.go:28] interesting pod/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 06 20:00:51.604289 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:51.604184 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 06 20:00:52.324882 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:52.324824 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:52.603271 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:52.603188 2585 patch_prober.go:28] interesting pod/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 06 20:00:52.603271 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:52.603254 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 06 20:00:53.603897 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:53.603861 2585 patch_prober.go:28] interesting pod/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 06 20:00:53.604288 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:53.603916 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 06 20:00:54.603024 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:54.602986 2585 patch_prober.go:28] interesting pod/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 06 20:00:54.603210 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:54.603047 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 06 20:00:55.607426 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:55.607391 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:00:55.608514 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:00:55.608493 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:01:01.028728 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.028694 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj"] Jul 06 20:01:01.031359 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.031340 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.035081 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.035058 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e1f716\"/\"openshift-service-ca.crt\"" Jul 06 20:01:01.035081 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.035063 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e1f716\"/\"kube-root-ca.crt\"" Jul 06 20:01:01.035245 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.035095 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1f716\"/\"auth-test-model-e1f716-kserve-self-signed-certs\"" Jul 06 20:01:01.035245 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.035132 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1f716\"/\"default-dockercfg-gsmj7\"" Jul 06 20:01:01.047211 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.047185 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj"] Jul 06 20:01:01.083461 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.083429 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tls-certs\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.083603 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.083472 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-home\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.083603 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.083494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tmp-dir\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.083603 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.083512 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-dshm\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.083603 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.083566 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wbfl\" (UniqueName: \"kubernetes.io/projected/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-kube-api-access-7wbfl\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.083777 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.083630 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-model-cache\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.184427 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tls-certs\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.184427 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-home\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.184695 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tmp-dir\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.184695 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184475 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-dshm\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.184695 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184499 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wbfl\" (UniqueName: \"kubernetes.io/projected/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-kube-api-access-7wbfl\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.184695 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184562 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-model-cache\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.184946 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184921 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tmp-dir\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.185008 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.184964 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-home\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.185079 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.185062 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-model-cache\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.186794 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.186764 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-dshm\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.187034 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.187018 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tls-certs\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.197582 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.197562 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wbfl\" (UniqueName: \"kubernetes.io/projected/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-kube-api-access-7wbfl\") pod \"auth-test-model-e1f716-kserve-5bdb76cf79-85bwj\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.342770 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.342648 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:01.489280 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:01.489223 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj"] Jul 06 20:01:01.491343 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:01:01.491314 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c48486_a8c9_4d76_ad8c_ba7b3d94fce9.slice/crio-6cf09f31eecd338beff69b0df6023600da8389140ddf93ebe4f8a21a7253c658 WatchSource:0}: Error finding container 6cf09f31eecd338beff69b0df6023600da8389140ddf93ebe4f8a21a7253c658: Status 404 returned error can't find the container with id 6cf09f31eecd338beff69b0df6023600da8389140ddf93ebe4f8a21a7253c658 Jul 06 20:01:02.460726 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:02.460689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" event={"ID":"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9","Type":"ContainerStarted","Data":"4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71"} Jul 06 20:01:02.461106 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:02.460735 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" event={"ID":"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9","Type":"ContainerStarted","Data":"6cf09f31eecd338beff69b0df6023600da8389140ddf93ebe4f8a21a7253c658"} Jul 06 20:01:02.485878 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:02.485828 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" podStartSLOduration=1.485813434 podStartE2EDuration="1.485813434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:02.485185768 +0000 UTC m=+2272.127420593" watchObservedRunningTime="2026-07-06 20:01:02.485813434 +0000 UTC m=+2272.128048292" Jul 06 20:01:11.343608 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:11.343533 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:11.343608 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:11.343579 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:11.356169 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:11.356150 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:11.510479 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:11.510456 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:01:28.942220 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:28.942182 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn"] Jul 06 20:01:28.944837 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:28.944817 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:28.947572 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:28.947553 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"openshift-service-ca.crt\"" Jul 06 20:01:28.948811 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:28.948788 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"default-dockercfg-5x755\"" Jul 06 20:01:28.948811 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:28.948806 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"auth-test-model-035267-kserve-self-signed-certs\"" Jul 06 20:01:28.948998 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:28.948812 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"kube-root-ca.crt\"" Jul 06 20:01:28.959301 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:28.959275 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn"] Jul 06 20:01:29.107456 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.107428 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cfd13-4584-498b-930d-1d47a61e4719-tls-certs\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.107626 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.107466 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-dshm\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.107626 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.107574 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-model-cache\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.107626 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.107599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-tmp-dir\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.107762 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.107626 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-home\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.107762 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.107732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvmb\" (UniqueName: \"kubernetes.io/projected/6d8cfd13-4584-498b-930d-1d47a61e4719-kube-api-access-7kvmb\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.208773 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.208700 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvmb\" (UniqueName: \"kubernetes.io/projected/6d8cfd13-4584-498b-930d-1d47a61e4719-kube-api-access-7kvmb\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.208773 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.208735 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cfd13-4584-498b-930d-1d47a61e4719-tls-certs\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.208773 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.208758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-dshm\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.209036 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.208823 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-model-cache\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.209036 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.208846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-tmp-dir\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.209036 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.208872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-home\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.209337 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.209301 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-model-cache\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.209430 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.209303 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-home\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.209430 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.209361 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-tmp-dir\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.211086 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.211062 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-dshm\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.211200 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.211182 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cfd13-4584-498b-930d-1d47a61e4719-tls-certs\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.219438 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.219416 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvmb\" (UniqueName: \"kubernetes.io/projected/6d8cfd13-4584-498b-930d-1d47a61e4719-kube-api-access-7kvmb\") pod \"auth-test-model-035267-kserve-59b4989d59-msqmn\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.254203 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.254179 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:29.383920 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.383888 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn"] Jul 06 20:01:29.387720 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:01:29.387691 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8cfd13_4584_498b_930d_1d47a61e4719.slice/crio-8682be5141ecd410ec37cc65dc1496e6e1ebbf134a85b3cdd6d3c72699559fe2 WatchSource:0}: Error finding container 8682be5141ecd410ec37cc65dc1496e6e1ebbf134a85b3cdd6d3c72699559fe2: Status 404 returned error can't find the container with id 8682be5141ecd410ec37cc65dc1496e6e1ebbf134a85b3cdd6d3c72699559fe2 Jul 06 20:01:29.577451 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.577422 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" event={"ID":"6d8cfd13-4584-498b-930d-1d47a61e4719","Type":"ContainerStarted","Data":"f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4"} Jul 06 20:01:29.577583 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.577463 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" event={"ID":"6d8cfd13-4584-498b-930d-1d47a61e4719","Type":"ContainerStarted","Data":"8682be5141ecd410ec37cc65dc1496e6e1ebbf134a85b3cdd6d3c72699559fe2"} Jul 06 20:01:29.604792 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:29.604747 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" podStartSLOduration=1.604731533 podStartE2EDuration="1.604731533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:29.602942355 +0000 UTC m=+2299.245177181" watchObservedRunningTime="2026-07-06 20:01:29.604731533 +0000 UTC m=+2299.246966358" Jul 06 20:01:39.254337 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:39.254294 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:39.254819 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:39.254351 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:39.267022 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:39.266996 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:01:39.632790 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:01:39.632716 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:03:36.738710 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:36.738617 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj"] Jul 06 20:03:37.624070 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:37.624037 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn"] Jul 06 20:03:44.531743 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.531713 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv"] Jul 06 20:03:44.534228 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.534211 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.536770 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.536746 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1f716\"/\"sub-test-model-e1f716-kserve-self-signed-certs\"" Jul 06 20:03:44.543723 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.543702 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv"] Jul 06 20:03:44.592592 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.592569 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-dshm\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.592717 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.592605 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c633e267-9926-439b-b3ca-bcc2c40684b9-tls-certs\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.592717 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.592633 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-home\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.592717 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.592648 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-model-cache\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.592889 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.592746 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-tmp-dir\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.594124 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.593208 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pd4l\" (UniqueName: \"kubernetes.io/projected/c633e267-9926-439b-b3ca-bcc2c40684b9-kube-api-access-2pd4l\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694359 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-dshm\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694539 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694373 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c633e267-9926-439b-b3ca-bcc2c40684b9-tls-certs\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694539 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-home\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694539 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-model-cache\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694539 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694472 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-tmp-dir\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694539 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pd4l\" (UniqueName: \"kubernetes.io/projected/c633e267-9926-439b-b3ca-bcc2c40684b9-kube-api-access-2pd4l\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694896 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694872 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-home\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694953 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694899 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-model-cache\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.694953 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.694935 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-tmp-dir\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.696624 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.696603 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-dshm\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.696996 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.696974 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c633e267-9926-439b-b3ca-bcc2c40684b9-tls-certs\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.704913 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.704893 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pd4l\" (UniqueName: \"kubernetes.io/projected/c633e267-9926-439b-b3ca-bcc2c40684b9-kube-api-access-2pd4l\") pod \"sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.843511 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.843449 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:44.979405 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:44.979375 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv"] Jul 06 20:03:44.981337 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:03:44.981301 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc633e267_9926_439b_b3ca_bcc2c40684b9.slice/crio-ba03f4b567fe6b0238f7fc8152e37c2c225b75cba23fe5a9f1284d58c1a7c32f WatchSource:0}: Error finding container ba03f4b567fe6b0238f7fc8152e37c2c225b75cba23fe5a9f1284d58c1a7c32f: Status 404 returned error can't find the container with id ba03f4b567fe6b0238f7fc8152e37c2c225b75cba23fe5a9f1284d58c1a7c32f Jul 06 20:03:45.183921 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:45.183836 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" event={"ID":"c633e267-9926-439b-b3ca-bcc2c40684b9","Type":"ContainerStarted","Data":"6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b"} Jul 06 20:03:45.183921 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:45.183885 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" event={"ID":"c633e267-9926-439b-b3ca-bcc2c40684b9","Type":"ContainerStarted","Data":"ba03f4b567fe6b0238f7fc8152e37c2c225b75cba23fe5a9f1284d58c1a7c32f"} Jul 06 20:03:45.202953 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:45.202910 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" podStartSLOduration=1.202897506 podStartE2EDuration="1.202897506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:45.202180854 +0000 UTC m=+2434.844415681" watchObservedRunningTime="2026-07-06 20:03:45.202897506 +0000 UTC m=+2434.845132331" Jul 06 20:03:51.758390 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:51.758329 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" podUID="12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" containerName="main" containerID="cri-o://4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71" gracePeriod=45 Jul 06 20:03:52.009046 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.008979 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:03:52.160815 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.160784 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-model-cache\") pod \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " Jul 06 20:03:52.160979 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.160829 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-dshm\") pod \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " Jul 06 20:03:52.160979 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.160850 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wbfl\" (UniqueName: \"kubernetes.io/projected/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-kube-api-access-7wbfl\") pod \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " Jul 06 20:03:52.160979 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.160875 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tls-certs\") pod \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " Jul 06 20:03:52.160979 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.160918 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-home\") pod \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " Jul 06 20:03:52.160979 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.160932 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tmp-dir\") pod \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\" (UID: \"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9\") " Jul 06 20:03:52.161251 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.161094 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-model-cache" (OuterVolumeSpecName: "model-cache") pod "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" (UID: "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.161251 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.161200 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-home" (OuterVolumeSpecName: "home") pod "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" (UID: "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.161386 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.161261 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" (UID: "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.163016 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.162995 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-dshm" (OuterVolumeSpecName: "dshm") pod "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" (UID: "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.163098 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.163080 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-kube-api-access-7wbfl" (OuterVolumeSpecName: "kube-api-access-7wbfl") pod "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" (UID: "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9"). InnerVolumeSpecName "kube-api-access-7wbfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:03:52.163144 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.163110 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" (UID: "12c48486-a8c9-4d76-ad8c-ba7b3d94fce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:03:52.220172 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.220145 2585 generic.go:356] "Generic (PLEG): container finished" podID="12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" containerID="4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71" exitCode=0 Jul 06 20:03:52.220289 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.220195 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" Jul 06 20:03:52.220289 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.220223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" event={"ID":"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9","Type":"ContainerDied","Data":"4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71"} Jul 06 20:03:52.220289 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.220268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj" event={"ID":"12c48486-a8c9-4d76-ad8c-ba7b3d94fce9","Type":"ContainerDied","Data":"6cf09f31eecd338beff69b0df6023600da8389140ddf93ebe4f8a21a7253c658"} Jul 06 20:03:52.220396 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.220291 2585 scope.go:117] "RemoveContainer" containerID="4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71" Jul 06 20:03:52.229223 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.229206 2585 scope.go:117] "RemoveContainer" containerID="4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71" Jul 06 20:03:52.229461 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:03:52.229442 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71\": container with ID starting with 4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71 not found: ID does not exist" containerID="4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71" Jul 06 20:03:52.229523 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.229470 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71"} err="failed to get container status \"4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71\": rpc error: code = NotFound desc = could not find container \"4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71\": container with ID starting with 4ce10fbadeb2f29e27d65af9bf0a86e74b8bbe2cfff0a8d293f0155192299b71 not found: ID does not exist" Jul 06 20:03:52.245127 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.245108 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj"] Jul 06 20:03:52.250921 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.250900 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/auth-test-model-e1f716-kserve-5bdb76cf79-85bwj"] Jul 06 20:03:52.262070 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.262027 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.262070 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.262044 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.262070 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.262053 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.262070 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.262061 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.262070 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.262068 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.262246 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.262076 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wbfl\" (UniqueName: \"kubernetes.io/projected/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9-kube-api-access-7wbfl\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.643912 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.643820 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" podUID="6d8cfd13-4584-498b-930d-1d47a61e4719" containerName="main" containerID="cri-o://f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4" gracePeriod=45 Jul 06 20:03:52.872405 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.872375 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" path="/var/lib/kubelet/pods/12c48486-a8c9-4d76-ad8c-ba7b3d94fce9/volumes" Jul 06 20:03:52.892475 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.892453 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:03:52.967020 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.966962 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-tmp-dir\") pod \"6d8cfd13-4584-498b-930d-1d47a61e4719\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " Jul 06 20:03:52.967020 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967004 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-model-cache\") pod \"6d8cfd13-4584-498b-930d-1d47a61e4719\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " Jul 06 20:03:52.967161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967026 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cfd13-4584-498b-930d-1d47a61e4719-tls-certs\") pod \"6d8cfd13-4584-498b-930d-1d47a61e4719\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " Jul 06 20:03:52.967161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967069 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-dshm\") pod \"6d8cfd13-4584-498b-930d-1d47a61e4719\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " Jul 06 20:03:52.967161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967108 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kvmb\" (UniqueName: \"kubernetes.io/projected/6d8cfd13-4584-498b-930d-1d47a61e4719-kube-api-access-7kvmb\") pod \"6d8cfd13-4584-498b-930d-1d47a61e4719\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " Jul 06 20:03:52.967161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967130 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-home\") pod \"6d8cfd13-4584-498b-930d-1d47a61e4719\" (UID: \"6d8cfd13-4584-498b-930d-1d47a61e4719\") " Jul 06 20:03:52.967373 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967236 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d8cfd13-4584-498b-930d-1d47a61e4719" (UID: "6d8cfd13-4584-498b-930d-1d47a61e4719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.967373 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967246 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d8cfd13-4584-498b-930d-1d47a61e4719" (UID: "6d8cfd13-4584-498b-930d-1d47a61e4719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.967373 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967343 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.967373 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967356 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:52.967548 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.967418 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-home" (OuterVolumeSpecName: "home") pod "6d8cfd13-4584-498b-930d-1d47a61e4719" (UID: "6d8cfd13-4584-498b-930d-1d47a61e4719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.969234 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.969203 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-dshm" (OuterVolumeSpecName: "dshm") pod "6d8cfd13-4584-498b-930d-1d47a61e4719" (UID: "6d8cfd13-4584-498b-930d-1d47a61e4719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:03:52.969339 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.969232 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8cfd13-4584-498b-930d-1d47a61e4719-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d8cfd13-4584-498b-930d-1d47a61e4719" (UID: "6d8cfd13-4584-498b-930d-1d47a61e4719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:03:52.969339 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:52.969273 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8cfd13-4584-498b-930d-1d47a61e4719-kube-api-access-7kvmb" (OuterVolumeSpecName: "kube-api-access-7kvmb") pod "6d8cfd13-4584-498b-930d-1d47a61e4719" (UID: "6d8cfd13-4584-498b-930d-1d47a61e4719"). InnerVolumeSpecName "kube-api-access-7kvmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:03:53.068577 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.068553 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:53.068694 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.068588 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvmb\" (UniqueName: \"kubernetes.io/projected/6d8cfd13-4584-498b-930d-1d47a61e4719-kube-api-access-7kvmb\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:53.068694 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.068604 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d8cfd13-4584-498b-930d-1d47a61e4719-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:53.068694 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.068617 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d8cfd13-4584-498b-930d-1d47a61e4719-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:03:53.225477 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.225404 2585 generic.go:356] "Generic (PLEG): container finished" podID="6d8cfd13-4584-498b-930d-1d47a61e4719" containerID="f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4" exitCode=0 Jul 06 20:03:53.225477 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.225450 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" Jul 06 20:03:53.225688 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.225496 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" event={"ID":"6d8cfd13-4584-498b-930d-1d47a61e4719","Type":"ContainerDied","Data":"f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4"} Jul 06 20:03:53.225688 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.225545 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn" event={"ID":"6d8cfd13-4584-498b-930d-1d47a61e4719","Type":"ContainerDied","Data":"8682be5141ecd410ec37cc65dc1496e6e1ebbf134a85b3cdd6d3c72699559fe2"} Jul 06 20:03:53.225688 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.225568 2585 scope.go:117] "RemoveContainer" containerID="f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4" Jul 06 20:03:53.234367 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.234352 2585 scope.go:117] "RemoveContainer" containerID="f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4" Jul 06 20:03:53.234610 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:03:53.234593 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4\": container with ID starting with f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4 not found: ID does not exist" containerID="f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4" Jul 06 20:03:53.234696 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.234620 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4"} err="failed to get container status \"f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4\": rpc error: code = NotFound desc = could not find container \"f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4\": container with ID starting with f6fc3ad579040442ad8ed493e15737a25a5d0e065af1089bc3429b69e649f1d4 not found: ID does not exist" Jul 06 20:03:53.254186 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.254164 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn"] Jul 06 20:03:53.256448 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:53.256429 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-035267/auth-test-model-035267-kserve-59b4989d59-msqmn"] Jul 06 20:03:54.843812 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:54.843780 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:54.844282 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:54.843822 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:54.856107 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:54.856088 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:03:54.873609 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:54.873585 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8cfd13-4584-498b-930d-1d47a61e4719" path="/var/lib/kubelet/pods/6d8cfd13-4584-498b-930d-1d47a61e4719/volumes" Jul 06 20:03:55.247248 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:03:55.247224 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:04:04.328688 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.328642 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr"] Jul 06 20:04:04.329061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.328953 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" containerName="main" Jul 06 20:04:04.329061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.328963 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" containerName="main" Jul 06 20:04:04.329061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.328979 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8cfd13-4584-498b-930d-1d47a61e4719" containerName="main" Jul 06 20:04:04.329061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.328984 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8cfd13-4584-498b-930d-1d47a61e4719" containerName="main" Jul 06 20:04:04.329061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.329035 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c48486-a8c9-4d76-ad8c-ba7b3d94fce9" containerName="main" Jul 06 20:04:04.329061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.329044 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8cfd13-4584-498b-930d-1d47a61e4719" containerName="main" Jul 06 20:04:04.332195 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.332178 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.335888 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.335867 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"kube-root-ca.crt\"" Jul 06 20:04:04.335888 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.335870 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"openshift-service-ca.crt\"" Jul 06 20:04:04.336069 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.335955 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"sub-test-model-035267-kserve-self-signed-certs\"" Jul 06 20:04:04.336069 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.335970 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"default-dockercfg-5x755\"" Jul 06 20:04:04.341819 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.341800 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr"] Jul 06 20:04:04.455536 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.455507 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-dshm\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.455702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.455544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-tmp-dir\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.455702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.455572 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21b1a27-8371-4701-9d60-b0da549faca8-tls-certs\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.455702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.455644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-home\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.455702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.455690 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmsf\" (UniqueName: \"kubernetes.io/projected/a21b1a27-8371-4701-9d60-b0da549faca8-kube-api-access-4hmsf\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.455877 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.455738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-model-cache\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557070 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21b1a27-8371-4701-9d60-b0da549faca8-tls-certs\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557274 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-home\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557274 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557122 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmsf\" (UniqueName: \"kubernetes.io/projected/a21b1a27-8371-4701-9d60-b0da549faca8-kube-api-access-4hmsf\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557274 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-model-cache\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557274 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-dshm\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557274 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-tmp-dir\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557637 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557611 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-home\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557798 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557637 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-tmp-dir\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.557798 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.557776 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-model-cache\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.560405 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.560375 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-dshm\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.560738 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.560716 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21b1a27-8371-4701-9d60-b0da549faca8-tls-certs\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.567818 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.567796 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmsf\" (UniqueName: \"kubernetes.io/projected/a21b1a27-8371-4701-9d60-b0da549faca8-kube-api-access-4hmsf\") pod \"sub-test-model-035267-kserve-6c8fcc9945-57jbr\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.641539 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.641475 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:04.776142 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:04.776118 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr"] Jul 06 20:04:04.777852 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:04:04.777821 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda21b1a27_8371_4701_9d60_b0da549faca8.slice/crio-2ed3f0ea98d3fdc9bafb65a5c1d7b3a8bbca7d0374002350ed41d5c3f37ba5c3 WatchSource:0}: Error finding container 2ed3f0ea98d3fdc9bafb65a5c1d7b3a8bbca7d0374002350ed41d5c3f37ba5c3: Status 404 returned error can't find the container with id 2ed3f0ea98d3fdc9bafb65a5c1d7b3a8bbca7d0374002350ed41d5c3f37ba5c3 Jul 06 20:04:05.279573 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:05.279535 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" event={"ID":"a21b1a27-8371-4701-9d60-b0da549faca8","Type":"ContainerStarted","Data":"d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962"} Jul 06 20:04:05.279760 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:05.279580 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" event={"ID":"a21b1a27-8371-4701-9d60-b0da549faca8","Type":"ContainerStarted","Data":"2ed3f0ea98d3fdc9bafb65a5c1d7b3a8bbca7d0374002350ed41d5c3f37ba5c3"} Jul 06 20:04:05.301865 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:05.301815 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" podStartSLOduration=1.301801268 podStartE2EDuration="1.301801268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:04:05.300888455 +0000 UTC m=+2454.943123272" watchObservedRunningTime="2026-07-06 20:04:05.301801268 +0000 UTC m=+2454.944036146" Jul 06 20:04:14.641882 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:14.641838 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:14.642333 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:14.641900 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:14.654638 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:14.654608 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:15.335784 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:15.335754 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:42.954278 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:42.954237 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv"] Jul 06 20:04:43.859544 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:43.859506 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr"] Jul 06 20:04:52.773281 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.773241 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz"] Jul 06 20:04:52.775869 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.775849 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.778996 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.778973 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e1f716\"/\"rate-test-model-e1f716-kserve-self-signed-certs\"" Jul 06 20:04:52.796246 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.796208 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz"] Jul 06 20:04:52.830453 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.830417 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5l7\" (UniqueName: \"kubernetes.io/projected/0b1798e8-6d21-4059-8315-c57c3952705c-kube-api-access-lk5l7\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.830642 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.830469 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-home\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.830642 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.830494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-model-cache\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.830642 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.830534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1798e8-6d21-4059-8315-c57c3952705c-tls-certs\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.830642 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.830564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-tmp-dir\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.830642 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.830598 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-dshm\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.931598 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.931557 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1798e8-6d21-4059-8315-c57c3952705c-tls-certs\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.931598 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.931603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-tmp-dir\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.931910 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.931623 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-dshm\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.931910 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.931754 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5l7\" (UniqueName: \"kubernetes.io/projected/0b1798e8-6d21-4059-8315-c57c3952705c-kube-api-access-lk5l7\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.931910 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.931827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-home\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.931910 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.931869 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-model-cache\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.932140 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.932080 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-tmp-dir\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.932213 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.932190 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-home\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.932249 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.932214 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-model-cache\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.934111 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.934091 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-dshm\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.934300 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.934282 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1798e8-6d21-4059-8315-c57c3952705c-tls-certs\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:52.949903 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:52.949869 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5l7\" (UniqueName: \"kubernetes.io/projected/0b1798e8-6d21-4059-8315-c57c3952705c-kube-api-access-lk5l7\") pod \"rate-test-model-e1f716-kserve-5b75655587-tskcz\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:53.084561 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:53.084463 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:04:53.226066 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:53.225931 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz"] Jul 06 20:04:53.228679 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:04:53.228622 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b1798e8_6d21_4059_8315_c57c3952705c.slice/crio-39437ea8373b27171bb59e58f1b0ae591cd9d6f5552f3637921a70c8af3bb95a WatchSource:0}: Error finding container 39437ea8373b27171bb59e58f1b0ae591cd9d6f5552f3637921a70c8af3bb95a: Status 404 returned error can't find the container with id 39437ea8373b27171bb59e58f1b0ae591cd9d6f5552f3637921a70c8af3bb95a Jul 06 20:04:53.487753 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:53.487720 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" event={"ID":"0b1798e8-6d21-4059-8315-c57c3952705c","Type":"ContainerStarted","Data":"05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6"} Jul 06 20:04:53.487933 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:53.487764 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" event={"ID":"0b1798e8-6d21-4059-8315-c57c3952705c","Type":"ContainerStarted","Data":"39437ea8373b27171bb59e58f1b0ae591cd9d6f5552f3637921a70c8af3bb95a"} Jul 06 20:04:53.513341 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:53.513281 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" podStartSLOduration=1.51326604 podStartE2EDuration="1.51326604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:04:53.512011821 +0000 UTC m=+2503.154246648" watchObservedRunningTime="2026-07-06 20:04:53.51326604 +0000 UTC m=+2503.155500865" Jul 06 20:04:57.973838 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:57.973794 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" podUID="c633e267-9926-439b-b3ca-bcc2c40684b9" containerName="main" containerID="cri-o://6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b" gracePeriod=45 Jul 06 20:04:58.223639 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.223614 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:04:58.380598 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.380567 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pd4l\" (UniqueName: \"kubernetes.io/projected/c633e267-9926-439b-b3ca-bcc2c40684b9-kube-api-access-2pd4l\") pod \"c633e267-9926-439b-b3ca-bcc2c40684b9\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " Jul 06 20:04:58.380814 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.380614 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-tmp-dir\") pod \"c633e267-9926-439b-b3ca-bcc2c40684b9\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " Jul 06 20:04:58.380814 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.380646 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-dshm\") pod \"c633e267-9926-439b-b3ca-bcc2c40684b9\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " Jul 06 20:04:58.380814 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.380730 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-home\") pod \"c633e267-9926-439b-b3ca-bcc2c40684b9\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " Jul 06 20:04:58.380814 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.380745 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-model-cache\") pod \"c633e267-9926-439b-b3ca-bcc2c40684b9\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " Jul 06 20:04:58.380814 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.380768 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c633e267-9926-439b-b3ca-bcc2c40684b9-tls-certs\") pod \"c633e267-9926-439b-b3ca-bcc2c40684b9\" (UID: \"c633e267-9926-439b-b3ca-bcc2c40684b9\") " Jul 06 20:04:58.381095 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.380937 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c633e267-9926-439b-b3ca-bcc2c40684b9" (UID: "c633e267-9926-439b-b3ca-bcc2c40684b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:58.381095 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.381037 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c633e267-9926-439b-b3ca-bcc2c40684b9" (UID: "c633e267-9926-439b-b3ca-bcc2c40684b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:58.381095 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.381053 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-home" (OuterVolumeSpecName: "home") pod "c633e267-9926-439b-b3ca-bcc2c40684b9" (UID: "c633e267-9926-439b-b3ca-bcc2c40684b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:58.382880 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.382855 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-dshm" (OuterVolumeSpecName: "dshm") pod "c633e267-9926-439b-b3ca-bcc2c40684b9" (UID: "c633e267-9926-439b-b3ca-bcc2c40684b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:58.382983 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.382876 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c633e267-9926-439b-b3ca-bcc2c40684b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c633e267-9926-439b-b3ca-bcc2c40684b9" (UID: "c633e267-9926-439b-b3ca-bcc2c40684b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:04:58.382983 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.382882 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c633e267-9926-439b-b3ca-bcc2c40684b9-kube-api-access-2pd4l" (OuterVolumeSpecName: "kube-api-access-2pd4l") pod "c633e267-9926-439b-b3ca-bcc2c40684b9" (UID: "c633e267-9926-439b-b3ca-bcc2c40684b9"). InnerVolumeSpecName "kube-api-access-2pd4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:04:58.481933 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.481909 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:58.481933 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.481933 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:58.482089 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.481941 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:58.482089 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.481949 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c633e267-9926-439b-b3ca-bcc2c40684b9-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:58.482089 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.481965 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c633e267-9926-439b-b3ca-bcc2c40684b9-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:58.482089 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.481972 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pd4l\" (UniqueName: \"kubernetes.io/projected/c633e267-9926-439b-b3ca-bcc2c40684b9-kube-api-access-2pd4l\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:58.510964 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.510934 2585 generic.go:356] "Generic (PLEG): container finished" podID="c633e267-9926-439b-b3ca-bcc2c40684b9" containerID="6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b" exitCode=0 Jul 06 20:04:58.511073 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.510975 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" Jul 06 20:04:58.511073 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.510999 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" event={"ID":"c633e267-9926-439b-b3ca-bcc2c40684b9","Type":"ContainerDied","Data":"6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b"} Jul 06 20:04:58.511073 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.511038 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv" event={"ID":"c633e267-9926-439b-b3ca-bcc2c40684b9","Type":"ContainerDied","Data":"ba03f4b567fe6b0238f7fc8152e37c2c225b75cba23fe5a9f1284d58c1a7c32f"} Jul 06 20:04:58.511073 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.511055 2585 scope.go:117] "RemoveContainer" containerID="6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b" Jul 06 20:04:58.519330 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.519309 2585 scope.go:117] "RemoveContainer" containerID="6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b" Jul 06 20:04:58.519580 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:04:58.519562 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b\": container with ID starting with 6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b not found: ID does not exist" containerID="6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b" Jul 06 20:04:58.519633 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.519589 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b"} err="failed to get container status \"6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b\": rpc error: code = NotFound desc = could not find container \"6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b\": container with ID starting with 6e7dc8a6e8f32971ab209980fae458aae7d2032eceb462c286644df4cad3c00b not found: ID does not exist" Jul 06 20:04:58.547045 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.547019 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv"] Jul 06 20:04:58.562887 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.562865 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/sub-test-model-e1f716-kserve-5548bbbf5f-lx2kv"] Jul 06 20:04:58.872562 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.872518 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c633e267-9926-439b-b3ca-bcc2c40684b9" path="/var/lib/kubelet/pods/c633e267-9926-439b-b3ca-bcc2c40684b9/volumes" Jul 06 20:04:58.878900 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:58.878862 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" podUID="a21b1a27-8371-4701-9d60-b0da549faca8" containerName="main" containerID="cri-o://d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962" gracePeriod=45 Jul 06 20:04:59.123998 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.123944 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:59.288744 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.288710 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-dshm\") pod \"a21b1a27-8371-4701-9d60-b0da549faca8\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " Jul 06 20:04:59.288937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.288779 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-model-cache\") pod \"a21b1a27-8371-4701-9d60-b0da549faca8\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " Jul 06 20:04:59.288937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.288824 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21b1a27-8371-4701-9d60-b0da549faca8-tls-certs\") pod \"a21b1a27-8371-4701-9d60-b0da549faca8\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " Jul 06 20:04:59.288937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.288848 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hmsf\" (UniqueName: \"kubernetes.io/projected/a21b1a27-8371-4701-9d60-b0da549faca8-kube-api-access-4hmsf\") pod \"a21b1a27-8371-4701-9d60-b0da549faca8\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " Jul 06 20:04:59.288937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.288872 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-home\") pod \"a21b1a27-8371-4701-9d60-b0da549faca8\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " Jul 06 20:04:59.288937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.288904 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-tmp-dir\") pod \"a21b1a27-8371-4701-9d60-b0da549faca8\" (UID: \"a21b1a27-8371-4701-9d60-b0da549faca8\") " Jul 06 20:04:59.289194 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.289039 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a21b1a27-8371-4701-9d60-b0da549faca8" (UID: "a21b1a27-8371-4701-9d60-b0da549faca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:59.289194 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.289142 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-home" (OuterVolumeSpecName: "home") pod "a21b1a27-8371-4701-9d60-b0da549faca8" (UID: "a21b1a27-8371-4701-9d60-b0da549faca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:59.289302 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.289202 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a21b1a27-8371-4701-9d60-b0da549faca8" (UID: "a21b1a27-8371-4701-9d60-b0da549faca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:59.289302 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.289222 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:59.290864 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.290838 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21b1a27-8371-4701-9d60-b0da549faca8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a21b1a27-8371-4701-9d60-b0da549faca8" (UID: "a21b1a27-8371-4701-9d60-b0da549faca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:04:59.290979 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.290880 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-dshm" (OuterVolumeSpecName: "dshm") pod "a21b1a27-8371-4701-9d60-b0da549faca8" (UID: "a21b1a27-8371-4701-9d60-b0da549faca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:04:59.290979 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.290932 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21b1a27-8371-4701-9d60-b0da549faca8-kube-api-access-4hmsf" (OuterVolumeSpecName: "kube-api-access-4hmsf") pod "a21b1a27-8371-4701-9d60-b0da549faca8" (UID: "a21b1a27-8371-4701-9d60-b0da549faca8"). InnerVolumeSpecName "kube-api-access-4hmsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:04:59.389853 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.389781 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:59.389853 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.389806 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a21b1a27-8371-4701-9d60-b0da549faca8-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:59.389853 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.389816 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmsf\" (UniqueName: \"kubernetes.io/projected/a21b1a27-8371-4701-9d60-b0da549faca8-kube-api-access-4hmsf\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:59.389853 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.389824 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:59.389853 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.389834 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a21b1a27-8371-4701-9d60-b0da549faca8-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:04:59.518640 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.518609 2585 generic.go:356] "Generic (PLEG): container finished" podID="a21b1a27-8371-4701-9d60-b0da549faca8" containerID="d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962" exitCode=0 Jul 06 20:04:59.518801 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.518677 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" Jul 06 20:04:59.518801 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.518703 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" event={"ID":"a21b1a27-8371-4701-9d60-b0da549faca8","Type":"ContainerDied","Data":"d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962"} Jul 06 20:04:59.518801 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.518743 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr" event={"ID":"a21b1a27-8371-4701-9d60-b0da549faca8","Type":"ContainerDied","Data":"2ed3f0ea98d3fdc9bafb65a5c1d7b3a8bbca7d0374002350ed41d5c3f37ba5c3"} Jul 06 20:04:59.518801 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.518761 2585 scope.go:117] "RemoveContainer" containerID="d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962" Jul 06 20:04:59.527736 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.527715 2585 scope.go:117] "RemoveContainer" containerID="d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962" Jul 06 20:04:59.528002 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:04:59.527983 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962\": container with ID starting with d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962 not found: ID does not exist" containerID="d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962" Jul 06 20:04:59.528061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.528009 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962"} err="failed to get container status \"d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962\": rpc error: code = NotFound desc = could not find container \"d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962\": container with ID starting with d2b946265002c6967ac9ef88c341e8f6c38a53886c36b7875725090fa85da962 not found: ID does not exist" Jul 06 20:04:59.559203 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.559172 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr"] Jul 06 20:04:59.568949 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:04:59.568922 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-035267/sub-test-model-035267-kserve-6c8fcc9945-57jbr"] Jul 06 20:05:00.872383 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:00.872350 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a21b1a27-8371-4701-9d60-b0da549faca8" path="/var/lib/kubelet/pods/a21b1a27-8371-4701-9d60-b0da549faca8/volumes" Jul 06 20:05:03.085299 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:03.085263 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:05:03.085722 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:03.085319 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:05:03.097614 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:03.097590 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:05:03.547872 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:03.547842 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:05:15.581482 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.581452 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj"] Jul 06 20:05:15.581902 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.581749 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c633e267-9926-439b-b3ca-bcc2c40684b9" containerName="main" Jul 06 20:05:15.581902 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.581760 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c633e267-9926-439b-b3ca-bcc2c40684b9" containerName="main" Jul 06 20:05:15.581902 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.581769 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21b1a27-8371-4701-9d60-b0da549faca8" containerName="main" Jul 06 20:05:15.581902 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.581775 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21b1a27-8371-4701-9d60-b0da549faca8" containerName="main" Jul 06 20:05:15.581902 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.581826 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21b1a27-8371-4701-9d60-b0da549faca8" containerName="main" Jul 06 20:05:15.581902 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.581834 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c633e267-9926-439b-b3ca-bcc2c40684b9" containerName="main" Jul 06 20:05:15.587038 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.587019 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.589614 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.589596 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"kube-root-ca.crt\"" Jul 06 20:05:15.589732 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.589596 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"rate-test-model-035267-kserve-self-signed-certs\"" Jul 06 20:05:15.590603 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.590587 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"default-dockercfg-5x755\"" Jul 06 20:05:15.590702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.590619 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-035267\"/\"openshift-service-ca.crt\"" Jul 06 20:05:15.616269 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.616246 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj"] Jul 06 20:05:15.630785 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.630765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-home\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.630923 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.630810 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9bxg\" (UniqueName: \"kubernetes.io/projected/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-kube-api-access-q9bxg\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.630923 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.630836 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tls-certs\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.631030 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.630926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-model-cache\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.631030 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.630975 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-dshm\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.631030 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.631008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tmp-dir\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731376 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9bxg\" (UniqueName: \"kubernetes.io/projected/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-kube-api-access-q9bxg\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731511 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731384 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tls-certs\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731511 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-model-cache\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731511 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731436 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-dshm\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731511 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731466 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tmp-dir\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731760 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731520 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-home\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731871 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731849 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-model-cache\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731941 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731883 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-home\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.731999 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.731957 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tmp-dir\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.733764 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.733744 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-dshm\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.733952 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.733934 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tls-certs\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.741479 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.741459 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9bxg\" (UniqueName: \"kubernetes.io/projected/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-kube-api-access-q9bxg\") pod \"rate-test-model-035267-kserve-6c8b558cd7-dvndj\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:15.895858 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:15.895758 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:16.030343 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:16.030319 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj"] Jul 06 20:05:16.031795 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:05:16.031772 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcece1c61_bc0c_42a6_baf6_d2bf4048d94a.slice/crio-c6de3df4e92ab991a632c7e25aaa951faa5d5a394fa5e3c264fa1142946c3f08 WatchSource:0}: Error finding container c6de3df4e92ab991a632c7e25aaa951faa5d5a394fa5e3c264fa1142946c3f08: Status 404 returned error can't find the container with id c6de3df4e92ab991a632c7e25aaa951faa5d5a394fa5e3c264fa1142946c3f08 Jul 06 20:05:16.590949 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:16.590916 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" event={"ID":"cece1c61-bc0c-42a6-baf6-d2bf4048d94a","Type":"ContainerStarted","Data":"3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f"} Jul 06 20:05:16.591331 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:16.590958 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" event={"ID":"cece1c61-bc0c-42a6-baf6-d2bf4048d94a","Type":"ContainerStarted","Data":"c6de3df4e92ab991a632c7e25aaa951faa5d5a394fa5e3c264fa1142946c3f08"} Jul 06 20:05:16.618088 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:16.618038 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" podStartSLOduration=1.618022093 podStartE2EDuration="1.618022093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:16.616994319 +0000 UTC m=+2526.259229142" watchObservedRunningTime="2026-07-06 20:05:16.618022093 +0000 UTC m=+2526.260256976" Jul 06 20:05:25.895916 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:25.895884 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:25.896344 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:25.895929 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:25.907909 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:25.907887 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:05:26.647431 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:05:26.647397 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:06:18.224193 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:18.224165 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz"] Jul 06 20:06:19.133739 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:19.133712 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj"] Jul 06 20:06:33.242905 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.242815 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" podUID="0b1798e8-6d21-4059-8315-c57c3952705c" containerName="main" containerID="cri-o://05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6" gracePeriod=45 Jul 06 20:06:33.493309 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.493249 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:06:33.580912 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.580877 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-dshm\") pod \"0b1798e8-6d21-4059-8315-c57c3952705c\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " Jul 06 20:06:33.580912 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.580912 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1798e8-6d21-4059-8315-c57c3952705c-tls-certs\") pod \"0b1798e8-6d21-4059-8315-c57c3952705c\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " Jul 06 20:06:33.581146 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.580944 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk5l7\" (UniqueName: \"kubernetes.io/projected/0b1798e8-6d21-4059-8315-c57c3952705c-kube-api-access-lk5l7\") pod \"0b1798e8-6d21-4059-8315-c57c3952705c\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " Jul 06 20:06:33.581146 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.580966 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-tmp-dir\") pod \"0b1798e8-6d21-4059-8315-c57c3952705c\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " Jul 06 20:06:33.581266 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.581155 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-home\") pod \"0b1798e8-6d21-4059-8315-c57c3952705c\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " Jul 06 20:06:33.581266 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.581199 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b1798e8-6d21-4059-8315-c57c3952705c" (UID: "0b1798e8-6d21-4059-8315-c57c3952705c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:33.581266 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.581211 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-model-cache\") pod \"0b1798e8-6d21-4059-8315-c57c3952705c\" (UID: \"0b1798e8-6d21-4059-8315-c57c3952705c\") " Jul 06 20:06:33.581491 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.581451 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-home" (OuterVolumeSpecName: "home") pod "0b1798e8-6d21-4059-8315-c57c3952705c" (UID: "0b1798e8-6d21-4059-8315-c57c3952705c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:33.581557 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.581489 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b1798e8-6d21-4059-8315-c57c3952705c" (UID: "0b1798e8-6d21-4059-8315-c57c3952705c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:33.581621 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.581563 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:33.581621 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.581610 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:33.583108 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.583079 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-dshm" (OuterVolumeSpecName: "dshm") pod "0b1798e8-6d21-4059-8315-c57c3952705c" (UID: "0b1798e8-6d21-4059-8315-c57c3952705c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:33.583201 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.583135 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1798e8-6d21-4059-8315-c57c3952705c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b1798e8-6d21-4059-8315-c57c3952705c" (UID: "0b1798e8-6d21-4059-8315-c57c3952705c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:33.583201 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.583147 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1798e8-6d21-4059-8315-c57c3952705c-kube-api-access-lk5l7" (OuterVolumeSpecName: "kube-api-access-lk5l7") pod "0b1798e8-6d21-4059-8315-c57c3952705c" (UID: "0b1798e8-6d21-4059-8315-c57c3952705c"). InnerVolumeSpecName "kube-api-access-lk5l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:33.682606 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.682577 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk5l7\" (UniqueName: \"kubernetes.io/projected/0b1798e8-6d21-4059-8315-c57c3952705c-kube-api-access-lk5l7\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:33.682606 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.682603 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:33.682783 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.682617 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1798e8-6d21-4059-8315-c57c3952705c-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:33.682783 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.682630 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1798e8-6d21-4059-8315-c57c3952705c-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:33.936196 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.936112 2585 generic.go:356] "Generic (PLEG): container finished" podID="0b1798e8-6d21-4059-8315-c57c3952705c" containerID="05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6" exitCode=0 Jul 06 20:06:33.936196 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.936154 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" Jul 06 20:06:33.936353 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.936200 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" event={"ID":"0b1798e8-6d21-4059-8315-c57c3952705c","Type":"ContainerDied","Data":"05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6"} Jul 06 20:06:33.936353 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.936236 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz" event={"ID":"0b1798e8-6d21-4059-8315-c57c3952705c","Type":"ContainerDied","Data":"39437ea8373b27171bb59e58f1b0ae591cd9d6f5552f3637921a70c8af3bb95a"} Jul 06 20:06:33.936353 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.936269 2585 scope.go:117] "RemoveContainer" containerID="05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6" Jul 06 20:06:33.944627 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.944612 2585 scope.go:117] "RemoveContainer" containerID="05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6" Jul 06 20:06:33.944878 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:06:33.944860 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6\": container with ID starting with 05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6 not found: ID does not exist" containerID="05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6" Jul 06 20:06:33.944930 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.944886 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6"} err="failed to get container status \"05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6\": rpc error: code = NotFound desc = could not find container \"05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6\": container with ID starting with 05e48172b8cec0b766527d58696642aab1cfc2bc2296b6a068987169754167c6 not found: ID does not exist" Jul 06 20:06:33.959613 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.959595 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz"] Jul 06 20:06:33.962515 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:33.962496 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e1f716/rate-test-model-e1f716-kserve-5b75655587-tskcz"] Jul 06 20:06:34.153105 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.153045 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" podUID="cece1c61-bc0c-42a6-baf6-d2bf4048d94a" containerName="main" containerID="cri-o://3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f" gracePeriod=45 Jul 06 20:06:34.400547 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.400525 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:06:34.488807 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.488774 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tls-certs\") pod \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " Jul 06 20:06:34.488807 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.488812 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9bxg\" (UniqueName: \"kubernetes.io/projected/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-kube-api-access-q9bxg\") pod \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " Jul 06 20:06:34.489056 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.488828 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-dshm\") pod \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " Jul 06 20:06:34.489056 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.488881 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-home\") pod \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " Jul 06 20:06:34.489056 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.488960 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-model-cache\") pod \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " Jul 06 20:06:34.489056 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.488999 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tmp-dir\") pod \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\" (UID: \"cece1c61-bc0c-42a6-baf6-d2bf4048d94a\") " Jul 06 20:06:34.489263 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.489232 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-model-cache" (OuterVolumeSpecName: "model-cache") pod "cece1c61-bc0c-42a6-baf6-d2bf4048d94a" (UID: "cece1c61-bc0c-42a6-baf6-d2bf4048d94a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:34.489320 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.489242 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-home" (OuterVolumeSpecName: "home") pod "cece1c61-bc0c-42a6-baf6-d2bf4048d94a" (UID: "cece1c61-bc0c-42a6-baf6-d2bf4048d94a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:34.489391 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.489310 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cece1c61-bc0c-42a6-baf6-d2bf4048d94a" (UID: "cece1c61-bc0c-42a6-baf6-d2bf4048d94a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:34.489430 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.489398 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-home\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:34.489430 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.489418 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-model-cache\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:34.491002 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.490974 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-dshm" (OuterVolumeSpecName: "dshm") pod "cece1c61-bc0c-42a6-baf6-d2bf4048d94a" (UID: "cece1c61-bc0c-42a6-baf6-d2bf4048d94a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:34.491131 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.491073 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cece1c61-bc0c-42a6-baf6-d2bf4048d94a" (UID: "cece1c61-bc0c-42a6-baf6-d2bf4048d94a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:34.491131 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.491119 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-kube-api-access-q9bxg" (OuterVolumeSpecName: "kube-api-access-q9bxg") pod "cece1c61-bc0c-42a6-baf6-d2bf4048d94a" (UID: "cece1c61-bc0c-42a6-baf6-d2bf4048d94a"). InnerVolumeSpecName "kube-api-access-q9bxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:34.590286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.590209 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tls-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:34.590286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.590234 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9bxg\" (UniqueName: \"kubernetes.io/projected/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-kube-api-access-q9bxg\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:34.590286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.590244 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-dshm\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:34.590286 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.590254 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cece1c61-bc0c-42a6-baf6-d2bf4048d94a-tmp-dir\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:06:34.872746 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.872647 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1798e8-6d21-4059-8315-c57c3952705c" path="/var/lib/kubelet/pods/0b1798e8-6d21-4059-8315-c57c3952705c/volumes" Jul 06 20:06:34.941847 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.941815 2585 generic.go:356] "Generic (PLEG): container finished" podID="cece1c61-bc0c-42a6-baf6-d2bf4048d94a" containerID="3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f" exitCode=0 Jul 06 20:06:34.942021 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.941857 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" Jul 06 20:06:34.942021 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.941903 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" event={"ID":"cece1c61-bc0c-42a6-baf6-d2bf4048d94a","Type":"ContainerDied","Data":"3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f"} Jul 06 20:06:34.942021 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.941941 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj" event={"ID":"cece1c61-bc0c-42a6-baf6-d2bf4048d94a","Type":"ContainerDied","Data":"c6de3df4e92ab991a632c7e25aaa951faa5d5a394fa5e3c264fa1142946c3f08"} Jul 06 20:06:34.942021 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.941963 2585 scope.go:117] "RemoveContainer" containerID="3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f" Jul 06 20:06:34.950269 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.950117 2585 scope.go:117] "RemoveContainer" containerID="3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f" Jul 06 20:06:34.950413 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:06:34.950395 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f\": container with ID starting with 3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f not found: ID does not exist" containerID="3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f" Jul 06 20:06:34.950464 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.950421 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f"} err="failed to get container status \"3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f\": rpc error: code = NotFound desc = could not find container \"3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f\": container with ID starting with 3b99e970edd39d9be732537085e4e443e02ecbcfc954c442f03b2f3331628f7f not found: ID does not exist" Jul 06 20:06:34.969532 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.969507 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj"] Jul 06 20:06:34.985457 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:34.985436 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-035267/rate-test-model-035267-kserve-6c8b558cd7-dvndj"] Jul 06 20:06:36.872938 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:06:36.872901 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cece1c61-bc0c-42a6-baf6-d2bf4048d94a" path="/var/lib/kubelet/pods/cece1c61-bc0c-42a6-baf6-d2bf4048d94a/volumes" Jul 06 20:07:04.539337 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:04.539303 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp"] Jul 06 20:07:04.539952 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:04.539521 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" containerID="cri-o://cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2" gracePeriod=30 Jul 06 20:07:06.005587 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:06.005549 2585 patch_prober.go:28] interesting pod/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:07:06.006002 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:06.005607 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:07:09.784206 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.784182 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:07:09.874445 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874358 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-envoy\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874445 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874412 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b18423-503b-4879-9763-329386c91c3a-istio-podinfo\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874454 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b18423-503b-4879-9763-329386c91c3a-istiod-ca-cert\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874470 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-certs\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874495 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-data\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874511 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tbfv\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-kube-api-access-2tbfv\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874528 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-istio-token\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874553 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-credential-socket\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.874702 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874631 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-socket\") pod \"86b18423-503b-4879-9763-329386c91c3a\" (UID: \"86b18423-503b-4879-9763-329386c91c3a\") " Jul 06 20:07:09.875061 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.874878 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:09.875175 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.875056 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:09.875175 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.875131 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:09.875175 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.875161 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b18423-503b-4879-9763-329386c91c3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:07:09.875579 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.875552 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:09.876750 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.876730 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b18423-503b-4879-9763-329386c91c3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:07:09.876908 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.876886 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:09.877132 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.877115 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:09.877204 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.877159 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-kube-api-access-2tbfv" (OuterVolumeSpecName: "kube-api-access-2tbfv") pod "86b18423-503b-4879-9763-329386c91c3a" (UID: "86b18423-503b-4879-9763-329386c91c3a"). InnerVolumeSpecName "kube-api-access-2tbfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:09.975408 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975370 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975408 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975401 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b18423-503b-4879-9763-329386c91c3a-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975408 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975411 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b18423-503b-4879-9763-329386c91c3a-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975408 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975420 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975707 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975428 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975707 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975436 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tbfv\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-kube-api-access-2tbfv\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975707 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975445 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b18423-503b-4879-9763-329386c91c3a-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975707 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975453 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:09.975707 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:09.975460 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b18423-503b-4879-9763-329386c91c3a-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:10.110296 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.110266 2585 generic.go:356] "Generic (PLEG): container finished" podID="86b18423-503b-4879-9763-329386c91c3a" containerID="cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2" exitCode=0 Jul 06 20:07:10.110446 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.110315 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" Jul 06 20:07:10.110446 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.110327 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" event={"ID":"86b18423-503b-4879-9763-329386c91c3a","Type":"ContainerDied","Data":"cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2"} Jul 06 20:07:10.110446 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.110369 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp" event={"ID":"86b18423-503b-4879-9763-329386c91c3a","Type":"ContainerDied","Data":"6baee27271365696790ea7ef49fabb479299131d738b62ce0abc2538554530fd"} Jul 06 20:07:10.110446 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.110389 2585 scope.go:117] "RemoveContainer" containerID="cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2" Jul 06 20:07:10.121352 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.121325 2585 scope.go:117] "RemoveContainer" containerID="d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af" Jul 06 20:07:10.128853 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.128836 2585 scope.go:117] "RemoveContainer" containerID="cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2" Jul 06 20:07:10.129093 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:07:10.129075 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2\": container with ID starting with cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2 not found: ID does not exist" containerID="cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2" Jul 06 20:07:10.129140 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.129102 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2"} err="failed to get container status \"cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2\": rpc error: code = NotFound desc = could not find container \"cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2\": container with ID starting with cbe3eaf3d0c2b0ba58f5d4ecfb72c5ec7b7841eb75caa575e58372cd6c0996b2 not found: ID does not exist" Jul 06 20:07:10.129140 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.129119 2585 scope.go:117] "RemoveContainer" containerID="d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af" Jul 06 20:07:10.129364 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:07:10.129345 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af\": container with ID starting with d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af not found: ID does not exist" containerID="d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af" Jul 06 20:07:10.129409 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.129369 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af"} err="failed to get container status \"d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af\": rpc error: code = NotFound desc = could not find container \"d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af\": container with ID starting with d6c7bd5943f020b94ba3f5bcc476e9963105098dd6aa51bfc489205ce36179af not found: ID does not exist" Jul 06 20:07:10.139073 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.139054 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp"] Jul 06 20:07:10.141825 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.141807 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e1f716-openshift-default-688687fd9f-7fblp"] Jul 06 20:07:10.873040 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:10.873006 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86b18423-503b-4879-9763-329386c91c3a" path="/var/lib/kubelet/pods/86b18423-503b-4879-9763-329386c91c3a/volumes" Jul 06 20:07:13.719068 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:13.719028 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m"] Jul 06 20:07:13.719530 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:13.719283 2585 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" containerID="cri-o://04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047" gracePeriod=30 Jul 06 20:07:14.416254 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:14.416224 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b64697b-dnjxw_b89b435a-1408-4ed2-8f88-bd2ad2f41caa/authorino/0.log" Jul 06 20:07:18.975506 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:18.975478 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:07:19.145174 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145067 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-credential-socket\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145174 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145106 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istiod-ca-cert\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145174 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145139 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-envoy\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145174 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145159 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-token\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145561 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145190 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-socket\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145561 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145266 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-podinfo\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145561 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145331 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpd4v\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-kube-api-access-wpd4v\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145561 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145395 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-data\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145561 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145428 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-certs\") pod \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\" (UID: \"c97b7aa8-0cc5-4c5c-983a-74607a28013e\") " Jul 06 20:07:19.145561 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145431 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:19.145915 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145630 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:19.145915 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145710 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:07:19.145915 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145815 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-credential-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.145915 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145838 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-socket\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.145915 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.145858 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:19.146509 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.146474 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:19.147913 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.147884 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:07:19.148036 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.147987 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:19.148036 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.147998 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:07:19.148112 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.148078 2585 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-kube-api-access-wpd4v" (OuterVolumeSpecName: "kube-api-access-wpd4v") pod "c97b7aa8-0cc5-4c5c-983a-74607a28013e" (UID: "c97b7aa8-0cc5-4c5c-983a-74607a28013e"). InnerVolumeSpecName "kube-api-access-wpd4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:07:19.150668 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.150626 2585 generic.go:356] "Generic (PLEG): container finished" podID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerID="04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047" exitCode=0 Jul 06 20:07:19.150780 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.150750 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" event={"ID":"c97b7aa8-0cc5-4c5c-983a-74607a28013e","Type":"ContainerDied","Data":"04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047"} Jul 06 20:07:19.150856 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.150836 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" event={"ID":"c97b7aa8-0cc5-4c5c-983a-74607a28013e","Type":"ContainerDied","Data":"dceebc2e3ac7fa78e9eea022beb918c0f08f095f127b8b8e97717a1682d1f49f"} Jul 06 20:07:19.150910 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.150847 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m" Jul 06 20:07:19.150910 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.150874 2585 scope.go:117] "RemoveContainer" containerID="04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047" Jul 06 20:07:19.167727 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.167702 2585 scope.go:117] "RemoveContainer" containerID="2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836" Jul 06 20:07:19.175832 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.175809 2585 scope.go:117] "RemoveContainer" containerID="04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047" Jul 06 20:07:19.176107 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:07:19.176087 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047\": container with ID starting with 04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047 not found: ID does not exist" containerID="04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047" Jul 06 20:07:19.176150 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.176117 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047"} err="failed to get container status \"04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047\": rpc error: code = NotFound desc = could not find container \"04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047\": container with ID starting with 04e31c2b440f2f406bf05cf36ec0ef65e6ee1abf6e69767f13145584207ae047 not found: ID does not exist" Jul 06 20:07:19.176150 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.176139 2585 scope.go:117] "RemoveContainer" containerID="2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836" Jul 06 20:07:19.176378 ip-10-0-131-106 kubenswrapper[2585]: E0706 20:07:19.176360 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836\": container with ID starting with 2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836 not found: ID does not exist" containerID="2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836" Jul 06 20:07:19.176420 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.176382 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836"} err="failed to get container status \"2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836\": rpc error: code = NotFound desc = could not find container \"2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836\": container with ID starting with 2235ddb0c17bf6cde728cc66926891bc297f407eb65b1814c065bb5046007836 not found: ID does not exist" Jul 06 20:07:19.193839 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.193810 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m"] Jul 06 20:07:19.201055 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.201031 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-035267-openshift-default-999cd8f55-5px2m"] Jul 06 20:07:19.247132 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.247101 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-workload-certs\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.247132 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.247131 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istiod-ca-cert\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.247132 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.247141 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-envoy\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.247378 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.247152 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-token\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.247378 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.247163 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-podinfo\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.247378 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.247174 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpd4v\" (UniqueName: \"kubernetes.io/projected/c97b7aa8-0cc5-4c5c-983a-74607a28013e-kube-api-access-wpd4v\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.247378 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.247189 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97b7aa8-0cc5-4c5c-983a-74607a28013e-istio-data\") on node \"ip-10-0-131-106.ec2.internal\" DevicePath \"\"" Jul 06 20:07:19.597937 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:19.597908 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-c4rh9_9e5ad88d-e037-4ad8-90c2-db6f9760a81c/cleanup/0.log" Jul 06 20:07:20.105973 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:20.105942 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mrp5j_d6c1bf2d-7302-4f07-99bb-db4a88b6351b/manager/1.log" Jul 06 20:07:20.384888 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:20.384810 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fcb8c7879-p8wwl_f2c5d423-34ef-4155-90b0-5244db05d22f/copy-manifests/0.log" Jul 06 20:07:20.403155 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:20.403126 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fcb8c7879-p8wwl_f2c5d423-34ef-4155-90b0-5244db05d22f/manager/0.log" Jul 06 20:07:20.871765 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:20.871728 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" path="/var/lib/kubelet/pods/c97b7aa8-0cc5-4c5c-983a-74607a28013e/volumes" Jul 06 20:07:22.092551 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:22.092519 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b64697b-dnjxw_b89b435a-1408-4ed2-8f88-bd2ad2f41caa/authorino/0.log" Jul 06 20:07:23.390011 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:23.389976 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-568d56697dbk4zs_2183335f-beb5-42ef-8199-137449dfb291/istio-proxy/0.log" Jul 06 20:07:23.650581 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:23.650508 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bf5b454b-q6pwb_b2095723-c283-4e1b-a9ce-4893b588b5b9/kube-auth-proxy/0.log" Jul 06 20:07:23.900908 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:23.900825 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5dfc694f5c-cdq4j_8a8f7a8d-d28e-45e5-be42-3137ba6e139b/istio-proxy/1.log" Jul 06 20:07:25.143875 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:25.143847 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ctjd2_8998fd01-b071-4a81-a58f-c06e90ed0c21/main/0.log" Jul 06 20:07:29.076122 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076078 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c7tz4/must-gather-wg58t"] Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076490 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1798e8-6d21-4059-8315-c57c3952705c" containerName="main" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076507 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1798e8-6d21-4059-8315-c57c3952705c" containerName="main" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076521 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076529 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076537 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cece1c61-bc0c-42a6-baf6-d2bf4048d94a" containerName="main" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076547 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cece1c61-bc0c-42a6-baf6-d2bf4048d94a" containerName="main" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076556 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076565 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076586 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" Jul 06 20:07:29.076632 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076594 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076685 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1798e8-6d21-4059-8315-c57c3952705c" containerName="main" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076698 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cece1c61-bc0c-42a6-baf6-d2bf4048d94a" containerName="main" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076709 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076717 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076727 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076736 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97b7aa8-0cc5-4c5c-983a-74607a28013e" containerName="istio-proxy" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076826 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" Jul 06 20:07:29.077161 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.076836 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b18423-503b-4879-9763-329386c91c3a" containerName="istio-proxy" Jul 06 20:07:29.081357 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.081329 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.083858 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.083839 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7tz4\"/\"openshift-service-ca.crt\"" Jul 06 20:07:29.083952 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.083876 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c7tz4\"/\"default-dockercfg-ph479\"" Jul 06 20:07:29.084973 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.084959 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7tz4\"/\"kube-root-ca.crt\"" Jul 06 20:07:29.097381 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.097357 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7tz4/must-gather-wg58t"] Jul 06 20:07:29.120990 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.120963 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37ee326d-ea31-4801-aa50-fcf55670e1f9-must-gather-output\") pod \"must-gather-wg58t\" (UID: \"37ee326d-ea31-4801-aa50-fcf55670e1f9\") " pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.121132 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.121022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9nl\" (UniqueName: \"kubernetes.io/projected/37ee326d-ea31-4801-aa50-fcf55670e1f9-kube-api-access-nw9nl\") pod \"must-gather-wg58t\" (UID: \"37ee326d-ea31-4801-aa50-fcf55670e1f9\") " pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.221824 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.221784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9nl\" (UniqueName: \"kubernetes.io/projected/37ee326d-ea31-4801-aa50-fcf55670e1f9-kube-api-access-nw9nl\") pod \"must-gather-wg58t\" (UID: \"37ee326d-ea31-4801-aa50-fcf55670e1f9\") " pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.222011 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.221868 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37ee326d-ea31-4801-aa50-fcf55670e1f9-must-gather-output\") pod \"must-gather-wg58t\" (UID: \"37ee326d-ea31-4801-aa50-fcf55670e1f9\") " pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.222181 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.222163 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/37ee326d-ea31-4801-aa50-fcf55670e1f9-must-gather-output\") pod \"must-gather-wg58t\" (UID: \"37ee326d-ea31-4801-aa50-fcf55670e1f9\") " pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.232205 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.232178 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9nl\" (UniqueName: \"kubernetes.io/projected/37ee326d-ea31-4801-aa50-fcf55670e1f9-kube-api-access-nw9nl\") pod \"must-gather-wg58t\" (UID: \"37ee326d-ea31-4801-aa50-fcf55670e1f9\") " pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.390289 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.390191 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7tz4/must-gather-wg58t" Jul 06 20:07:29.534942 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.534909 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7tz4/must-gather-wg58t"] Jul 06 20:07:29.542266 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:29.542250 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:07:30.198296 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:30.198256 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7tz4/must-gather-wg58t" event={"ID":"37ee326d-ea31-4801-aa50-fcf55670e1f9","Type":"ContainerStarted","Data":"8be3700cf4ebfc3fdf93b8497156fe6e2217a3701afbb68536f3eecc13157f64"} Jul 06 20:07:31.204767 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:31.204734 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7tz4/must-gather-wg58t" event={"ID":"37ee326d-ea31-4801-aa50-fcf55670e1f9","Type":"ContainerStarted","Data":"4db45848823aa42cdbc1d206118c5026d70f3be1ee229783eefa24de65750297"} Jul 06 20:07:31.205185 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:31.204778 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7tz4/must-gather-wg58t" event={"ID":"37ee326d-ea31-4801-aa50-fcf55670e1f9","Type":"ContainerStarted","Data":"4fd072ee76b9a3d45194c269817c97f3f5d0479fe368f393cfb51495f77eb635"} Jul 06 20:07:31.228327 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:31.228274 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c7tz4/must-gather-wg58t" podStartSLOduration=1.381223676 podStartE2EDuration="2.228258169s" totalImagesPullingTime="847.034493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:31.225464118 +0000 UTC m=+2660.867698944" watchObservedRunningTime="2026-07-06 20:07:31.228258169 +0000 UTC m=+2660.870492996" Jul 06 20:07:32.274009 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:32.273967 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6mcrp_86b026e4-7a94-40ad-bae2-22ae1a89d7a7/global-pull-secret-syncer/0.log" Jul 06 20:07:32.530957 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:32.530858 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vkf9p_e7c3880b-47bb-41bd-a6f6-d2eadcc62027/konnectivity-agent/0.log" Jul 06 20:07:32.557637 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:32.557609 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-106.ec2.internal_188ef7b029f22b3628e6f2183332d139/haproxy/0.log" Jul 06 20:07:37.637017 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:37.636639 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b64697b-dnjxw_b89b435a-1408-4ed2-8f88-bd2ad2f41caa/authorino/0.log" Jul 06 20:07:40.559677 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:40.559628 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjbtw_6cc288b3-1a2c-452d-8ff6-44e9d4db2d19/node-exporter/0.log" Jul 06 20:07:40.594957 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:40.594922 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjbtw_6cc288b3-1a2c-452d-8ff6-44e9d4db2d19/kube-rbac-proxy/0.log" Jul 06 20:07:40.631346 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:40.631315 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjbtw_6cc288b3-1a2c-452d-8ff6-44e9d4db2d19/init-textfile/0.log" Jul 06 20:07:41.859342 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.859303 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r"] Jul 06 20:07:41.866017 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.865989 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:41.878686 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.878441 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r"] Jul 06 20:07:41.944833 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.944487 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-sys\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:41.944833 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.944549 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-podres\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:41.944833 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.944598 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-proc\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:41.944833 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.944637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbtwz\" (UniqueName: \"kubernetes.io/projected/46b79936-3310-42d6-bbb6-d0b70159c8b2-kube-api-access-fbtwz\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:41.944833 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:41.944710 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-lib-modules\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.045908 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.045861 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-lib-modules\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046124 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.045948 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-sys\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046124 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.045994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-podres\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046124 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.046028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-proc\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046124 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.046040 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-sys\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046124 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.046065 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbtwz\" (UniqueName: \"kubernetes.io/projected/46b79936-3310-42d6-bbb6-d0b70159c8b2-kube-api-access-fbtwz\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046124 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.046112 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-proc\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046360 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.046142 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-lib-modules\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.046752 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.046726 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46b79936-3310-42d6-bbb6-d0b70159c8b2-podres\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.058469 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.058419 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbtwz\" (UniqueName: \"kubernetes.io/projected/46b79936-3310-42d6-bbb6-d0b70159c8b2-kube-api-access-fbtwz\") pod \"perf-node-gather-daemonset-cfv5r\" (UID: \"46b79936-3310-42d6-bbb6-d0b70159c8b2\") " pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.181099 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.180752 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:42.367277 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.366806 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r"] Jul 06 20:07:42.368934 ip-10-0-131-106 kubenswrapper[2585]: W0706 20:07:42.368896 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46b79936_3310_42d6_bbb6_d0b70159c8b2.slice/crio-c17f2bedc6c31faa3cd0695cf5951fed10b36c7667e0c104e8bcb61677a5816d WatchSource:0}: Error finding container c17f2bedc6c31faa3cd0695cf5951fed10b36c7667e0c104e8bcb61677a5816d: Status 404 returned error can't find the container with id c17f2bedc6c31faa3cd0695cf5951fed10b36c7667e0c104e8bcb61677a5816d Jul 06 20:07:42.857211 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:42.857181 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-z6dqc_9c6aae22-4e4c-4149-8495-85a0265dbafd/networking-console-plugin/0.log" Jul 06 20:07:43.278004 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:43.277956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" event={"ID":"46b79936-3310-42d6-bbb6-d0b70159c8b2","Type":"ContainerStarted","Data":"596e99ee20634a3f59f935c64fea7c1dc37f7e1c451fdb0aff2645a186d79ce8"} Jul 06 20:07:43.278477 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:43.278028 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:43.278477 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:43.278052 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" event={"ID":"46b79936-3310-42d6-bbb6-d0b70159c8b2","Type":"ContainerStarted","Data":"c17f2bedc6c31faa3cd0695cf5951fed10b36c7667e0c104e8bcb61677a5816d"} Jul 06 20:07:43.300084 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:43.300033 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" podStartSLOduration=2.300016361 podStartE2EDuration="2.300016361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:43.299898684 +0000 UTC m=+2672.942133521" watchObservedRunningTime="2026-07-06 20:07:43.300016361 +0000 UTC m=+2672.942251186" Jul 06 20:07:45.947032 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:45.947003 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbkxs_9aa6b80e-8165-4bcb-a8bf-306c48480540/dns/0.log" Jul 06 20:07:45.973832 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:45.973791 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbkxs_9aa6b80e-8165-4bcb-a8bf-306c48480540/kube-rbac-proxy/0.log" Jul 06 20:07:46.134172 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:46.134143 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wrf9_1483d7bc-f311-4b6e-857b-14c0189746d9/dns-node-resolver/0.log" Jul 06 20:07:46.847697 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:46.847647 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-59ptz_e265b652-468c-46c6-8d9e-8c48ae290882/node-ca/0.log" Jul 06 20:07:48.072671 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:48.072624 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-568d56697dbk4zs_2183335f-beb5-42ef-8199-137449dfb291/istio-proxy/0.log" Jul 06 20:07:48.308625 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:48.308597 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bf5b454b-q6pwb_b2095723-c283-4e1b-a9ce-4893b588b5b9/kube-auth-proxy/0.log" Jul 06 20:07:48.379599 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:48.379517 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5dfc694f5c-cdq4j_8a8f7a8d-d28e-45e5-be42-3137ba6e139b/istio-proxy/0.log" Jul 06 20:07:48.413708 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:48.413677 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5dfc694f5c-cdq4j_8a8f7a8d-d28e-45e5-be42-3137ba6e139b/istio-proxy/1.log" Jul 06 20:07:49.295774 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:49.295732 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c7tz4/perf-node-gather-daemonset-cfv5r" Jul 06 20:07:49.515770 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:49.515740 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-227hc_9a0256f5-2b8f-4737-96da-772a3f8dc561/serve-healthcheck-canary/0.log" Jul 06 20:07:50.259711 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:50.259666 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drf7c_b74cae87-3597-419b-803c-d0e4b93c861e/kube-rbac-proxy/0.log" Jul 06 20:07:50.282689 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:50.282646 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drf7c_b74cae87-3597-419b-803c-d0e4b93c861e/exporter/0.log" Jul 06 20:07:50.310186 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:50.310134 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drf7c_b74cae87-3597-419b-803c-d0e4b93c861e/extractor/0.log" Jul 06 20:07:52.167023 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:52.166990 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-568d56697dbk4zs_2183335f-beb5-42ef-8199-137449dfb291/istio-proxy/0.log" Jul 06 20:07:52.380853 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:52.380827 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bf5b454b-q6pwb_b2095723-c283-4e1b-a9ce-4893b588b5b9/kube-auth-proxy/0.log" Jul 06 20:07:52.435095 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:52.435024 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5dfc694f5c-cdq4j_8a8f7a8d-d28e-45e5-be42-3137ba6e139b/istio-proxy/0.log" Jul 06 20:07:52.470326 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:52.470298 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5dfc694f5c-cdq4j_8a8f7a8d-d28e-45e5-be42-3137ba6e139b/istio-proxy/1.log" Jul 06 20:07:53.459453 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:53.459388 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-c4rh9_9e5ad88d-e037-4ad8-90c2-db6f9760a81c/cleanup/0.log" Jul 06 20:07:53.799287 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:53.799254 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mrp5j_d6c1bf2d-7302-4f07-99bb-db4a88b6351b/manager/0.log" Jul 06 20:07:53.808809 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:53.808783 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mrp5j_d6c1bf2d-7302-4f07-99bb-db4a88b6351b/manager/1.log" Jul 06 20:07:53.822060 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:53.822034 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-568d56697dbk4zs_2183335f-beb5-42ef-8199-137449dfb291/istio-proxy/0.log" Jul 06 20:07:54.061761 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:54.061737 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54bf5b454b-q6pwb_b2095723-c283-4e1b-a9ce-4893b588b5b9/kube-auth-proxy/0.log" Jul 06 20:07:54.105940 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:54.105906 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5dfc694f5c-cdq4j_8a8f7a8d-d28e-45e5-be42-3137ba6e139b/istio-proxy/0.log" Jul 06 20:07:54.113515 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:54.113481 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fcb8c7879-p8wwl_f2c5d423-34ef-4155-90b0-5244db05d22f/manager/0.log" Jul 06 20:07:54.140529 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:54.140492 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fcb8c7879-p8wwl_f2c5d423-34ef-4155-90b0-5244db05d22f/copy-manifests/0.log" Jul 06 20:07:54.142826 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:54.142805 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5dfc694f5c-cdq4j_8a8f7a8d-d28e-45e5-be42-3137ba6e139b/istio-proxy/1.log" Jul 06 20:07:55.641130 ip-10-0-131-106 kubenswrapper[2585]: I0706 20:07:55.641104 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cf88bd784-bl46t_948b1022-c227-4641-bb08-3b955d81702b/manager/0.log"